1 2 Previous Next 29 Replies Latest reply on Mar 18, 2009 7:50 AM by manik

    is it possible for MVCCInvocationContext to have null mvccTC

      I have a JBC 3 cache instance configured to use a JTA transactionManager. Assuming that I have configured the cache instance properly, is it possible for a MVCCInvocationContext NOT to have reference to MVCCTransactionContext while thread of execution is inside MVCCInvocationContext.lookUpNode(fqn)?

      Specifically, a thread is inside MVCCInvocationContext.lookUpNode(fqn) and mvccTx is (sometimes) null. See sample call stack below.

      If this is NOT a valid state, does that mean that JBC isn't "seeing" my JTA txManager - ie I have a config problem?

      If it is a valid state, please explain why so.

      Here's a call stack:


      org.jboss.cache.invocation.MVCCInvocationContext.lookUpNode(MVCCInvocationContext.java:64)
      org.jboss.cache.commands.write.PutKeyValueCommand.perform(PutKeyValueCommand.java:78)
      org.jboss.cache.interceptors.CallInterceptor.invokeCommand(CallInterceptor.java:108)
      org.jboss.cache.interceptors.CallInterceptor.handleAlterCacheMethod(CallInterceptor.java:173)
      org.jboss.cache.interceptors.CallInterceptor.visitPutForExternalReadCommand(CallInterceptor.java:137)
      org.jboss.cache.commands.write.PutForExternalReadCommand.acceptVisitor(PutForExternalReadCommand.java:72)
      org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
      org.jboss.cache.interceptors.MVCCLockingInterceptor.handlePutForExternalReadCommand(MVCCLockingInterceptor.java:110)
      org.jboss.cache.interceptors.base.PrePostProcessingCommandInterceptor.visitPutForExternalReadCommand(PrePostProcessingCommandInterceptor.java:101)
      org.jboss.cache.commands.write.PutForExternalReadCommand.acceptVisitor(PutForExternalReadCommand.java:72)
      org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
      org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
      org.jboss.cache.commands.AbstractVisitor.visitPutForExternalReadCommand(AbstractVisitor.java:70)
      org.jboss.cache.commands.write.PutForExternalReadCommand.acceptVisitor(PutForExternalReadCommand.java:72)
      org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
      org.jboss.cache.interceptors.TxInterceptor.attachGtxAndPassUpChain(TxInterceptor.java:261)
      org.jboss.cache.interceptors.TxInterceptor.handleDefault(TxInterceptor.java:250)
      org.jboss.cache.commands.AbstractVisitor.visitPutForExternalReadCommand(AbstractVisitor.java:70)
      org.jboss.cache.commands.write.PutForExternalReadCommand.acceptVisitor(PutForExternalReadCommand.java:72)
      org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
      org.jboss.cache.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:108)
      org.jboss.cache.interceptors.CacheMgmtInterceptor.visitPutForExternalReadCommand(CacheMgmtInterceptor.java:102)
      org.jboss.cache.commands.write.PutForExternalReadCommand.acceptVisitor(PutForExternalReadCommand.java:72)
      org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
      org.jboss.cache.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:178)
      org.jboss.cache.interceptors.InvocationContextInterceptor.visitPutForExternalReadCommand(InvocationContextInterceptor.java:76)
      org.jboss.cache.commands.write.PutForExternalReadCommand.acceptVisitor(PutForExternalReadCommand.java:72)
      org.jboss.cache.interceptors.InterceptorChain.invoke(InterceptorChain.java:287)
      org.jboss.cache.invocation.CacheInvocationDelegate.putForExternalRead(CacheInvocationDelegate.java:459)
      org.hibernate.cache.jbc2.util.CacheHelper.putForExternalRead(CacheHelper.java:295)
      org.hibernate.cache.jbc2.util.CacheHelper.putForExternalRead(CacheHelper.java:268)
      org.hibernate.cache.jbc2.access.TransactionalAccessDelegate.putFromLoad(TransactionalAccessDelegate.java:80)
      org.hibernate.cache.jbc2.collection.TransactionalAccess.putFromLoad(TransactionalAccess.java:98)
      org.hibernate.engine.loading.CollectionLoadContext.addCollectionToCache(CollectionLoadContext.java:333)
      org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollection(CollectionLoadContext.java:279)
      org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollections(CollectionLoadContext.java:245)
      org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollections(CollectionLoadContext.java:218)
      org.hibernate.loader.Loader.endCollectionLoad(Loader.java:900)
      org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:888)
      org.hibernate.loader.Loader.doQuery(Loader.java:752)
      org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
      org.hibernate.loader.Loader.loadCollection(Loader.java:2015)
      org.hibernate.loader.collection.CollectionLoader.initialize(CollectionLoader.java:59)
      org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:587)
      org.hibernate.event.def.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:83)
      org.hibernate.impl.SessionImpl.initializeCollection(SessionImpl.java:1743)
      org.hibernate.collection.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:366)
      org.hibernate.collection.AbstractPersistentCollection.read(AbstractPersistentCollection.java:108)
      org.hibernate.collection.AbstractPersistentCollection.readSize(AbstractPersistentCollection.java:131)
      org.hibernate.collection.PersistentSet.isEmpty(PersistentSet.java:169)
      com.doppelganger.domain.User.getCurrentUserSession(User.java:306)
      com.doppelganger.framework.cache.LocalCacheTest$1.doInTransactionWithException(LocalCacheTest.java:62)
      com.doppelganger.test.service.impl.DgTxAnnotationDependingTemplate.executeWithException(DgTxAnnotationDependingTemplate.java:22)
      sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      java.lang.reflect.Method.invoke(Method.java:597)
      org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
      org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
      org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
      org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
      org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
      org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:77)
      com.doppelganger.service.tx.TxRetryAspect.doRetryIfNeeded(TxRetryAspect.java:114)
      com.doppelganger.service.tx.TxRetryAspect.retryIfNeededForTxAnnotation(TxRetryAspect.java:90)
      sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      java.lang.reflect.Method.invoke(Method.java:597)
      org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627)
      org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616)
      org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:64)
      org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:160)
      org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
      org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
      org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
      $Proxy182.executeWithException
      com.doppelganger.framework.cache.LocalCacheTest.testPutForExternalPutRemoveInTx1_readInTx2(LocalCacheTest.java:56)
      sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      java.lang.reflect.Method.invoke(Method.java:597)
      junit.framework.TestCase.runTest(TestCase.java:168)
      junit.framework.TestCase.runBare(TestCase.java:134)
      org.springframework.test.ConditionalTestCase.runBare(ConditionalTestCase.java:76)
      junit.framework.TestResult$1.protect(TestResult.java:110)


      thanks
      -nikita



        • 1. Re: is it possible for MVCCInvocationContext to have null mv

          Sorry, I meant 'mvccTCtx' from:

           public NodeSPI lookUpNode(Fqn fqn)
           {
           if (mvccTCtx != null) return mvccTCtx.lookUpNode(fqn);
           return lookedUpNodes == null ? null : lookedUpNodes.get(fqn);
           }
          


          • 2. Re: is it possible for MVCCInvocationContext to have null mv

            Well, it appears that the above *is* possible - but I wonder whether that is a bug. Basically, because of failSilently==true putForExternalRead ends up adding Node to MVCCInvocationContext.lookedUpNodes but subsequent evictNode() uses MVCCInvocationContext.mvccTCtx to look up the same node (and fails to find it)

            Setup: Hibernate, JBC 3.0.3.GA; MVCC; JTA tm injected; REPEATABLE_READ iso

            Consider the following scenario:
            pre: cache is empty

            tx.begin()
            cache.putForExternalRead(fqnA, val);
            cache.evictNode(fqnA) //expect eviction to succeed
            tx.commit()

            1. By the time invocationContextInteceptor invokes next interceptor, mvccInvocationContext.mvccTCtx is null!

            Given failSilently=true in putForExternalRead:

            getInvocationContext().getOptionOverrides().setFailSilently(true);
             getInvocationContext().getOptionOverrides().setForceAsynchronous(true);
             PutForExternalReadCommand command = commandsFactory.buildPutForExternalReadCommand(null, fqn, key, value);
             invoker.invoke(ctx, command);

            ... InvocationContextInterceptor sets transactionalContext twice in handleAll (at lines 150 and 164):

            if (txManager != null)
             {
             Transaction tx = getTransaction();
             GlobalTransaction realGtx = getGlobalTransaction(tx, gtx);
             if (tx == null && realGtx != null && realGtx.isRemote()) tx = txTable.getLocalTransaction(gtx);
             setTransactionalContext(tx, realGtx, null, ctx);
             }


            ..and


            if (optionOverride != null && optionOverride.isFailSilently())
             {
             log.debug("FAIL_SILENTLY Option is present - suspending any ongoing transaction.");
             suppressExceptions = true;
             if (ctx.getTransaction() != null)
             {
             suspendedTransaction = txManager.suspend();
             setTransactionalContext(null, null, null, ctx);
             if (trace) log.trace("Suspending transaction " + suspendedTransaction);
             resumeSuspended = true;
             }


            The first invocation of setTransactionalContext injects valid TransactionContext, Transaction into MVCCInvocationContext. but the second invocation clears them, such that invocationContext.getTransaction() will return null henceforth.

            Subsequently other interceptors are invoked, including TxInterceptor, whose attachGtxAndPassUpChain does not attach anything because ctx.getTransaction()==null. For the remainder of this invocation MVCCInvocationContext.mvccTCtx remains null and ctx.lookedUpNodes is used.

            2. Cache.evictNode fails to evict because ctx.lookUpNode looks in mvccTCtx and does not find Node.

            Subsequent Cache.evictNode(fqn) does not use failSilently option, so setTransactionalContext in InvocationContextInterceptor is NOT called twice and valid transactionContext remains injected into invocationContext. So, when ctx.lookUpNode() is called from EvictCommand.perform, existing ctx.mvccTCtx is consulted (and NOT 'lookUpNodes'). And thus, the node is not found, and eviction is not performed. NOTE: seems like any command that uses ctx.lookUpNode and is NOT invoked with failSilently==true will fail to lookup this node for the same reason.

            Also note that in NotifierImpl.resetInvocationContext lookedUpNodes are copied, so the above not-evicted-node would be copied as well.

            Thoughts?

            If the does seem like a bug, the obvious solution seems to be not to use failSilently option in Cache.putForExternalRead (and in other places?). What would be the implication?



            • 3. Re: is it possible for MVCCInvocationContext to have null mv
              manik

              Hi and sorry for not responding sooner.

              putForExternalRead was designed to specifically work outside the scope of any ongoing transaction, so that it does not cause any ongoing transaction to fail and roll back if the pFER() call fails.

              So this is why calls to ctx.lookUpNode() within the scope of a pFER() call uses the looked up nodes map on the InvocationContext rather than the TransactionContext, this is by design.

              The evict call, that then attempts to evict the node, should be referring to the transaction context and not the invocation context. This again is by design, and since the node does not exist in the transaction context, it would be loaded into the transaction context in MVCCLockingInterceptor.handleEvictFqnCommand().

              If for some reason your call to cache.evict() fails, then this is probably a bug.

              • 4. Re: is it possible for MVCCInvocationContext to have null mv

                Exactly, in my environment I am seeing precisely that: cache.evict() fails to evict. I wrote up what I thought was the root cause for of this failure: pfer writes to one (non-tx) context, while evict looks in another (please see my post right above yours)

                Could you please point out where in above reasoning I may be making a mistake (obviously being new to JBC I don't quite understand the internals yet)?

                fwiw, when I rebuilt jbc with pfer NOT using failSilently, evict-after-pfer started to work.

                However, it is true that thus far I am unable to replicate this failure in a jbc test suite test - so it is quite possible that the problem is not with JBC Core but, for example, with my setup or hibernate-jbosscache2 integration.

                thanks
                -nikita

                • 5. Re: is it possible for MVCCInvocationContext to have null mv

                  To follow up further. Below is a test i wrote trying to capture above failure. When I run it I observe the following:
                  - if no debugger is used, test passes
                  - if I step through with a debugger when EvictCommand.perform calls ctx.lookUpNode(fqn) the underlying MVCCInvocationContext sometimes has a non-null mvccTCtx and sometimes mvccTCtx is null?! It seems to depend on where I break and for how long - though pattern is unclear.

                  When mvccTCtx is null, then Node in question *is* found in lookedUpNodes and eviction proceeds.

                  When mvccTCtx is NOT null, it does not contain the node in question -> ctx.lookUpNode returns null (lookedUpNodes isn't consulted if mvccTCtx is null) and eviction does not proceed (command.perform returns true immediately)

                  Questions:

                  1. From my understanding ctx.mvccTCtx should NOT be null when used accessed from EvictCommand.perform in the above test. Do you agree? If not, why (how is it that mvccTCtx can be null in this case)?

                  2. Since it seems I can affect the outcome of the test merely by attaching a debugger and breaking throughput, I am guessing either multiple threads or timestamping is involved. Given that the entire duration of the test didn't exceed 40-50 seconds (including my think time) when varying results were obtained, this does not seem to be related to underlying JTA tx timing out. Any idea where to look for debugger breaks can affect?

                  thanks
                  -nikita

                  package org.jboss.cache.api;
                  
                  import org.jboss.cache.config.Configuration;
                  import org.jboss.cache.config.Configuration.NodeLockingScheme;
                  import org.jboss.cache.factories.UnitTestConfigurationFactory;
                  import org.testng.annotations.Test;
                  
                  import javax.transaction.TransactionManager;
                  
                  import org.jboss.cache.*;
                  import org.jboss.cache.lock.IsolationLevel;
                  
                  /**
                   * Tests that a node that was putFromExternalRead and then removed in TX1 does NOT
                   * get returned in subsequent TX2.
                   *
                   * @author nikita_tovstoles@mba.berkeley.edu
                   * @since 3.0.3.GA
                   */
                  @Test(groups = {"functional", "pessimistic"}, sequential = true, testName = "api.RemovedNodeResurrectionInSubsequentTxTest")
                  public class RemovedNodeResurrectionInSubsequentTxTest extends AbstractSingleCacheTest {
                  
                   private static final Fqn A_B = Fqn.fromString("/a/b");
                   private static final Fqn A = Fqn.fromString("/a");
                   private static final Fqn A_C = Fqn.fromString("/a/c");
                   private static final String KEY = "key";
                   private static final String VALUE = "value";
                   private static final String K2 = "k2";
                   private static final String V2 = "v2";
                   protected NodeSPI root;
                   protected TransactionManager txManager;
                  
                   public CacheSPI createCache() {
                   CacheSPI myCache = (CacheSPI<Object, Object>) new UnitTestCacheFactory<Object, Object>().createCache(UnitTestConfigurationFactory.createConfiguration(Configuration.CacheMode.LOCAL, true), false, getClass());
                   myCache.getConfiguration().setCacheMode(Configuration.CacheMode.LOCAL);
                   myCache.getConfiguration().setCacheLoaderConfig(null);
                   myCache.getConfiguration().setNodeLockingScheme(NodeLockingScheme.MVCC);
                   configure(myCache.getConfiguration());
                   myCache.start();
                   root = myCache.getRoot();
                   txManager = myCache.getTransactionManager();
                  
                   return myCache;
                   }
                  
                   protected void configure(Configuration c) {
                   c.setIsolationLevel(IsolationLevel.REPEATABLE_READ);
                   }
                  
                   public void testPferAndEvictInSameTx() throws Exception {
                  
                   Object val = null;
                  
                   cache.getRoot().addChild(A);
                  
                   txManager.begin();
                  
                   //put in cache
                   cache.putForExternalRead(A, KEY, VALUE);
                  
                   //evict from cache
                   cache.evict(A); //sometimes MVCCInvocationContext has ref to mvccTCtx, sometimes NOT
                  
                   //verify eviction
                   val = cache.get(A, KEY);
                   assert val == null : "get() after evict() returned " + val;
                  
                   txManager.commit();
                  
                   }
                  }
                  


                  • 6. Re: is it possible for MVCCInvocationContext to have null mv
                    manik

                    Thanks for the test.

                    "dukehoops" wrote:
                    To follow up further. Below is a test i wrote trying to capture above failure. When I run it I observe the following:
                    - if no debugger is used, test passes
                    - if I step through with a debugger when EvictCommand.perform calls ctx.lookUpNode(fqn) the underlying MVCCInvocationContext sometimes has a non-null mvccTCtx and sometimes mvccTCtx is null?! It seems to depend on where I break and for how long - though pattern is unclear.


                    Wow, that's not nice.

                    "dukehoops" wrote:

                    1. From my understanding ctx.mvccTCtx should NOT be null when used accessed from EvictCommand.perform in the above test. Do you agree? If not, why (how is it that mvccTCtx can be null in this case)?


                    Agreed. Between the InvocationContextInterceptor and the TxInterceptor, this should be made available.

                    "dukehoops" wrote:

                    2. Since it seems I can affect the outcome of the test merely by attaching a debugger and breaking throughput, I am guessing either multiple threads or timestamping is involved. Given that the entire duration of the test didn't exceed 40-50 seconds (including my think time) when varying results were obtained, this does not seem to be related to underlying JTA tx timing out. Any idea where to look for debugger breaks can affect?


                    It could be that the reference to the tCtx needs to be volatile, let me think about how and when this is accessed and let you know.

                    • 7. Re: is it possible for MVCCInvocationContext to have null mv
                      manik

                      Hmm, I don't see this failure on your test, even when running it on multiple threads, etc. Are you seeing this issue in a live use case, or just when using your debugger?

                      • 8. Re: is it possible for MVCCInvocationContext to have null mv

                        Manik,

                        This test is an attempt to distill an issue that only shows up under concurrent load. Please see the last page of this topic background info (pull up a chair - this may take awhile ;-):

                        http://www.jboss.org/index.html?module=bb&op=viewtopic&t=151524&postdays=0&postorder=asc&start=20

                        • 9. Re: is it possible for MVCCInvocationContext to have null mv

                          and yes, I started the above-linked topic because the problem described there is consistently repro-able on our system under mild concurrency.

                          • 10. Re: is it possible for MVCCInvocationContext to have null mv

                            aha!

                            So there *is* another thread at work - at least in this test environment. Look in createCache():

                            createCache(UnitTestConfigurationFactory.createConfiguration(Configuration.CacheMode.LOCAL, true); //true==some eviction policy

                            That's why when I break in ctx.lookUpNode, I see the following stack running on a thread called "EvictionTimer-0". this certainly explains why stepping through with a debugger alters results:


                            org.jboss.cache.invocation.MVCCInvocationContext.lookUpNode(MVCCInvocationContext.java:76)
                            org.jboss.cache.commands.write.EvictCommand.lookupForEviction(EvictCommand.java:168)
                            org.jboss.cache.commands.write.EvictCommand.perform(EvictCommand.java:93)
                            org.jboss.cache.interceptors.CallInterceptor.invokeCommand(CallInterceptor.java:108)
                            org.jboss.cache.interceptors.CallInterceptor.handleDefault(CallInterceptor.java:99)
                            org.jboss.cache.commands.AbstractVisitor.visitEvictFqnCommand(AbstractVisitor.java:85)
                            org.jboss.cache.commands.write.EvictCommand.acceptVisitor(EvictCommand.java:173)
                            org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
                            org.jboss.cache.interceptors.EvictionInterceptor.visitEvictFqnCommand(EvictionInterceptor.java:80)
                            org.jboss.cache.commands.write.EvictCommand.acceptVisitor(EvictCommand.java:173)
                            org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
                            org.jboss.cache.interceptors.MVCCLockingInterceptor.handleEvictFqnCommand(MVCCLockingInterceptor.java:141)
                            org.jboss.cache.interceptors.base.PrePostProcessingCommandInterceptor.visitEvictFqnCommand(PrePostProcessingCommandInterceptor.java:185)
                            org.jboss.cache.commands.write.EvictCommand.acceptVisitor(EvictCommand.java:173)
                            org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
                            org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
                            org.jboss.cache.commands.AbstractVisitor.visitEvictFqnCommand(AbstractVisitor.java:85)
                            org.jboss.cache.commands.write.EvictCommand.acceptVisitor(EvictCommand.java:173)
                            org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
                            org.jboss.cache.interceptors.TxInterceptor.attachGtxAndPassUpChain(TxInterceptor.java:261)
                            org.jboss.cache.interceptors.TxInterceptor.handleDefault(TxInterceptor.java:250)
                            org.jboss.cache.commands.AbstractVisitor.visitEvictFqnCommand(AbstractVisitor.java:85)
                            org.jboss.cache.commands.write.EvictCommand.acceptVisitor(EvictCommand.java:173)
                            org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
                            org.jboss.cache.interceptors.CacheMgmtInterceptor.visitEvictFqnCommand(CacheMgmtInterceptor.java:65)
                            org.jboss.cache.commands.write.EvictCommand.acceptVisitor(EvictCommand.java:173)
                            org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
                            org.jboss.cache.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:178)
                            org.jboss.cache.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:130)
                            org.jboss.cache.commands.AbstractVisitor.visitEvictFqnCommand(AbstractVisitor.java:85)
                            org.jboss.cache.commands.write.EvictCommand.acceptVisitor(EvictCommand.java:173)
                            org.jboss.cache.interceptors.InterceptorChain.invoke(InterceptorChain.java:287)
                            org.jboss.cache.invocation.CacheInvocationDelegate.evict(CacheInvocationDelegate.java:370)
                            org.jboss.cache.invocation.CacheInvocationDelegate.evict(CacheInvocationDelegate.java:374)
                            org.jboss.cache.eviction.DefaultEvictionActionPolicy.evict(DefaultEvictionActionPolicy.java:50)
                            org.jboss.cache.eviction.BaseEvictionAlgorithm.evictCacheNode(BaseEvictionAlgorithm.java:265)
                            org.jboss.cache.eviction.LRUAlgorithm.evict(LRUAlgorithm.java:121)
                            org.jboss.cache.eviction.LRUAlgorithm.prune(LRUAlgorithm.java:153)
                            org.jboss.cache.eviction.BaseEvictionAlgorithm.process(BaseEvictionAlgorithm.java:146)
                            org.jboss.cache.RegionImpl.processEvictionQueues(RegionImpl.java:128)
                            org.jboss.cache.eviction.EvictionTimerTask.handleRegion(EvictionTimerTask.java:115)
                            org.jboss.cache.eviction.EvictionTimerTask.processRegions(EvictionTimerTask.java:107)
                            org.jboss.cache.eviction.EvictionTimerTask.access$100(EvictionTimerTask.java:42)
                            org.jboss.cache.eviction.EvictionTimerTask$Task.run(EvictionTimerTask.java:133)
                            java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
                            java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
                            java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
                            java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
                            java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
                            java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
                            java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                            java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                            java.lang.Thread.run(Thread.java:619)


                            Now, i'll try to see what happens if I don't use an eviction policy (as, i think, is the case in our app)..

                            • 11. Re: is it possible for MVCCInvocationContext to have null mv

                              Well, it appears there isn't a problem in this test after all. To answer my own question above, mvccTCtx gets populated with Node in question in MVCCNodeHelper.wrapNodeForWriting (called by MVCCLockingInterceptor before EvictCommand.perform is executed).

                              Side note: Configuration.writeSkewCheck defaults to false (and not true as specified in JBC3 User Guide reference section Ch 12, p 104).

                              So the concurrency problem described in this post is probably rooted elsewhere :-(

                              http://www.jboss.org/index.html?module=bb&op=viewtopic&t=151524&postdays=0&postorder=asc&start=20

                              • 12. Re: is it possible for MVCCInvocationContext to have null mv

                                Got it. The following sequence will fail:

                                1. get()
                                2. pfer()
                                3. get() //fail here

                                However, if the 1st step is eliminated, second get succeeds:
                                pfer();
                                get();//success.

                                Here's the test:

                                package org.jboss.cache.api;
                                
                                import org.jboss.cache.config.Configuration;
                                import org.jboss.cache.config.Configuration.NodeLockingScheme;
                                import org.jboss.cache.factories.UnitTestConfigurationFactory;
                                import org.testng.annotations.Test;
                                
                                import javax.transaction.TransactionManager;
                                
                                import org.jboss.cache.*;
                                import org.jboss.cache.lock.IsolationLevel;
                                
                                /**
                                 * Tests that a node that was putFromExternalRead and then removed in TX1 does NOT
                                 * get returned in subsequent TX2.
                                 *
                                 * @author nikita_tovstoles@mba.berkeley.edu
                                 * @since 3.0.3.GA
                                 */
                                @Test(groups = {"functional", "pessimistic"}, sequential = true, testName = "api.RemovedNodeResurrectionInSubsequentTxTest")
                                public class RemovedNodeResurrectionInSubsequentTxTest extends AbstractSingleCacheTest {
                                
                                 private static final Fqn A_B = Fqn.fromString("/a/b");
                                 private static final Fqn A = Fqn.fromString("/a");
                                 private static final Fqn A_C = Fqn.fromString("/a/c");
                                 private static final String KEY = "key";
                                 private static final String VALUE = "value";
                                 private static final String K2 = "k2";
                                 private static final String V2 = "v2";
                                 protected NodeSPI root;
                                 protected TransactionManager txManager;
                                
                                 public CacheSPI createCache() {
                                 CacheSPI myCache = (CacheSPI<Object, Object>) new UnitTestCacheFactory<Object, Object>().createCache(UnitTestConfigurationFactory.createConfiguration(Configuration.CacheMode.LOCAL, false), false, getClass());
                                 myCache.getConfiguration().setCacheMode(Configuration.CacheMode.LOCAL);
                                 myCache.getConfiguration().setCacheLoaderConfig(null);
                                 myCache.getConfiguration().setNodeLockingScheme(NodeLockingScheme.MVCC);
                                 configure(myCache.getConfiguration());
                                 myCache.start();
                                 root = myCache.getRoot();
                                 txManager = myCache.getTransactionManager();
                                
                                 return myCache;
                                 }
                                
                                 protected void configure(Configuration c) {
                                 c.setIsolationLevel(IsolationLevel.REPEATABLE_READ);
                                 }
                                
                                 public void testPferAndEvictInSameTx() throws Exception {
                                
                                 Object val = null;
                                
                                 cache.getRoot().addChild(A);
                                
                                 txManager.begin();
                                
                                 val = cache.get(A_B, KEY); //N1 IF THIS LINE IS EXECUTED, TEST FAILS BELOW AT N2
                                
                                 //put in cache
                                 cache.putForExternalRead(A_B, KEY, VALUE);
                                
                                 val = cache.get(A_B, KEY);
                                 assert val != null : "get() after pfer() returned " + val; //N2 THIS WILL FAIL IF LINE N1 (ABOVE) IS EXECUTED (NOT COMMENTED OUT)
                                
                                 //evict from cache
                                 cache.evict(A_B); //sometimes MVCCInvocationContext has ref to mvccTCtx, sometimes NOT
                                
                                 //verify eviction
                                 val = cache.get(A_B, KEY);
                                 assert val == null : "get() after evict() returned " + val;
                                
                                 txManager.commit();
                                
                                 }
                                }
                                


                                Here's the output:




                                java.lang.AssertionError: get() after pfer() returned null
                                at org.jboss.cache.api.RemovedNodeResurrectionInSubsequentTxTest.testPferAndEvictInSameTx(RemovedNodeResurrectionInSubsequentTxTest.java:65)
                                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                                at java.lang.Thread.run(Thread.java:619)


                                However, if line marked as "N1" is commented out, test passes. Any ideas?

                                • 13. Re: is it possible for MVCCInvocationContext to have null mv
                                  manik

                                   

                                  "dukehoops" wrote:
                                  Side note: Configuration.writeSkewCheck defaults to false (and not true as specified in JBC3 User Guide reference section Ch 12, p 104).


                                  Well spotted. Thanks, corrected in trunk (3.1.0)

                                  • 14. Re: is it possible for MVCCInvocationContext to have null mv
                                    manik

                                    Had a look at your test, and your test is invalid. Here's why:

                                    Case 1: You have commented out the line marked N1. This works fine, since you first do a pFER() and then a get(). Expected behaviour.

                                    Case 2: Line marked N1 is not commented out. Here is what happens:

                                    1. tx starts
                                    2. read the value, which is null
                                    3. do a pFER(). Remember that pFER(), to be fail-safe, runs in a separate transaction. So this will suspend the tx started in 1, do the pFER(), and then resume the tx in 1.
                                    4. read the value again, you get a null. This is correct, since you have REPEATABLE_READ specified.

                                    Since line 3 happens outside the scope of the transaction, the transaction correctly treats this as a different, concurrent transaction modifying the value. And since you have already read the value as null, you will always see it as null for the scope of the transaction.

                                    As proof, try running your test either without the tx.begin() and tx.commit(), or specify a weaker isolation level like READ_COMMITTED.

                                    1 2 Previous Next