5 Replies Latest reply on Sep 14, 2007 8:13 AM by manik

    Problem with concurent put/remove operation.

    jacek187

      There is a problem with concurent put/remove operation in latest 1.4.1SP4 Cayenne version. When one thread is only removing and second only putting values into this same cache region, sometime put operation fails with exception:

      org.jboss.cache.NodeNotExistsException: node /a/b/c/d not found (gtx=GlobalTransaction::2, caller=Thread[Thread-0,5,main])
      at org.jboss.cache.TreeCache._put(TreeCache.java:4618)
      at org.jboss.cache.TreeCache._put(TreeCache.java:4579)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      at java.lang.reflect.Method.invoke(Unknown Source)
      at org.jgroups.blocks.MethodCall.invoke(MethodCall.java:330)
      at org.jboss.cache.interceptors.CallInterceptor.invoke(CallInterceptor.java:52)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.PessimisticLockInterceptor.invoke(PessimisticLockInterceptor.java:89)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      (...)
      Thread 1 is calling:

       cache.remove("/a/b/c/d");
       cache.remove("/a");
      

      and Thread 2 is calling:
       cache.putObject("/a/b/c/d", "text" + x);
      



      This same operations sequence fails too on latest 2.0.0.GA Habanero version but with another message:

      org.jboss.cache.pojo.PojoCacheException: detach /a/b/c/d failed
      at org.jboss.cache.pojo.impl.PojoCacheImpl.detach(PojoCacheImpl.java:158)
      at org.jboss.cache.pojo.impl.PojoCacheDelegate.putObjectII(PojoCacheDelegate.java:143)
      at org.jboss.cache.pojo.impl.PojoCacheImpl.putObject(PojoCacheImpl.java:136)
      at org.jboss.cache.pojo.impl.PojoCacheImpl.attach(PojoCacheImpl.java:101)
      at org.jboss.cache.pojo.impl.PojoCacheImpl.attach(PojoCacheImpl.java:93)
      at org.jboss.cache.aop.ConcurentOperationTest$WorkThread.run(ConcurentOperationTest.java:80)
      Caused by: org.jboss.cache.lock.UpgradeException: failure upgrading lock: fqn=/__JBossInternal__/a/b/c/d/_ID_/5c4o02w-youoh8-f5frdt5j-1-f5frdt5j-2, caller=GlobalTransaction:<192.168.0.104:4850>:6, lock=read owners=[GlobalTransaction:<192.168.0.104:4850>:5, GlobalTransaction:<192.168.0.104:4850>:6] (activeReaders=1, activeWriter=null, waitingReaders=0, waitingWriters=1, waitingUpgrader=1)
      at org.jboss.cache.lock.IdentityLock.acquire(IdentityLock.java:508)
      at org.jboss.cache.interceptors.PessimisticLockInterceptor$LockManager.acquire(PessimisticLockInterceptor.java:579)
      (...)

        • 1. Re: Problem with concurent put/remove operation.
          manik

          Are you using the pojo or core cache? Your 2.0.0 stack trace seems to imply pojo but your 1.4.1 stack trace seems to imply core...

          • 2. Re: Problem with concurent put/remove operation.
            jacek187

            I have tried many combinations: core, pojo and mix
            * imply core - test fails
            * imply pojo on 1.4.1 - test OK
            * impy pojo on 2.0.0 - test fails, so bug http://jira.jboss.com/jira/browse/JBCACHE-1166?page=all was created
            * mix plain and pojo - putObject/remove combination - test fails so bug http://jira.jboss.com/jira/browse/JBCACHE-1164?page=all was created.

            • 3. Re: Problem with concurent put/remove operation.
              manik

              Thanks - I've commented on the specfic JIRAs.

              What do you mean by test fails on core? (I presume this is 2.0.0 core?) Do you have a stack trace for this? The 2.0.0 stack trace above has to do with pojo.

              • 4. Re: Problem with concurent put/remove operation.
                jacek187

                impy core on 1.4.1 - described in http://www.jboss.com/index.html?module=bb&op=viewtopic&t=116337 forum thread
                I have just tested core on 2.0.0. GA - and exception was ... timeout.

                stack:
                org.jboss.cache.lock.TimeoutException: failure acquiring lock: fqn=/a/b/c/d, caller=GlobalTransaction::4, lock=write owner=GlobalTransaction::3 (activeReaders=0, activeWriter=Thread[Thread-2,5,main], waitingReaders=0, waitingWriters=0, waitingUpgrader=0)
                at org.jboss.cache.lock.IdentityLock.acquire(IdentityLock.java:528)
                at org.jboss.cache.interceptors.PessimisticLockInterceptor$LockManager.acquire(PessimisticLockInterceptor.java:579)
                at org.jboss.cache.interceptors.PessimisticLockInterceptor.acquireNodeLock(PessimisticLockInterceptor.java:393)
                at org.jboss.cache.interceptors.PessimisticLockInterceptor.lock(PessimisticLockInterceptor.java:329)
                at org.jboss.cache.interceptors.PessimisticLockInterceptor.invoke(PessimisticLockInterceptor.java:187)
                at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:76)
                at org.jboss.cache.interceptors.NotificationInterceptor.invoke(NotificationInterceptor.java:32)
                (...)
                main loop:

                 for (int x = 0; x < 1000; x++) {
                 tm.begin();
                 if (remove) {
                 cache.remove("/a/b/c/d");
                 cache.remove("/a");
                 } else {
                 cache.put("/a/b/c/d", "key","text" + x);
                 }
                 tm.commit();
                 Thread.sleep(1);
                 }
                


                one thread is putting key-value pair into /a/b/c/d node and second thread is removing this node.

                And now I don't known what is it: just bug or normal situation.
                If this is normal situation - how TimeoutException can be explained? One thread is executing only one operation in one transaction (classical deadlock needs 2 operations in one transaction...)

                I need to run many paraller threads, and every thread should modify exactly this same node. Which cache should be used? [I think that every cache (pojo, core, 1.4.1, 2.0.0 can be used)] Now looks that only PojoCache in version 1.4.1 works correctly in Multithread environment...

                • 5. Re: Problem with concurent put/remove operation.
                  manik

                  Note to anyone else following this thread: see http://jboss.org/index.html?module=bb&op=viewtopic&t=118186 for detailed analysis