1 Reply Latest reply on Sep 27, 2006 9:53 AM by jbirkenmaier

    Two nodes updating cache concurrently using a PojoCacheMBean

    jbirkenmaier

      I have a 3 node cluster and one node goes away. The two remaining nodes receive notice at exactly the same time and update their cache accordingly. Each is basically removing references to the disappeared node from cache. The problem occurs when the commit is done on the 2 nodes. Each one has a lock that the other node wants. Here is the exception:

      Caused by: java.lang.IllegalStateException: there is already a writer holding the lock: GlobalTransaction:<192.168.69.253:41197>:65
      at org.jboss.cache.lock.LockMap.setWriterIfNotNull(LockMap.java:96)
      at org.jboss.cache.lock.IdentityLock.acquireWriteLock(IdentityLock.java:204)
      at org.jboss.cache.Node.acquireWriteLock(Node.java:431)
      at org.jboss.cache.Node.acquire(Node.java:386)
      at org.jboss.cache.interceptors.OptimisticLockingInterceptor.lockNodes(OptimisticLockingInterceptor.java:149)
      at org.jboss.cache.interceptors.OptimisticLockingInterceptor.invoke(OptimisticLockingInterceptor.java:76)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.OptimisticReplicationInterceptor.invoke(OptimisticReplicationInterceptor.java:74)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.TxInterceptor.runPreparePhase(TxInterceptor.java:804)
      at org.jboss.cache.interceptors.TxInterceptor$LocalSynchronizationHandler.beforeCompletion(TxInterceptor.java:1069)
      ... 83 more
      07:13:13,767 ERROR [ResourceReplicatorBase] (JBoss Shutdown Hook) removeResourceDescriptor: org.jboss.tm.JBossRollbackException: Unable to commit, tx=TransactionImpl:XidImpl[FormatId=257, GlobalId=dragoneyes/119, BranchQual=, localId=119] status=STATUS_NO_TRANSACTION; - nested throwable: (java.lang.RuntimeException: ), retry the operation, current iteration: 0

      Each node throws this exception with an IP and port of the other node. I have retry code in there to retry my operations but when this exception is thrown, the cache becomes corrupt and I am unable to remove the data (again) from the cache. This works perfectly if one node gets there first but when they hit the cache at the same time....trouble.

        • 1. Re: Two nodes updating cache concurrently using a PojoCacheM
          jbirkenmaier

          Update: I have been tinkering with the code and removed the transaction that surrounds my cache updating and the exceptions have gone away (no surprise there). However, there is still a big problem. The two remaining nodes still update their own cache (with the hope that the updates will be replicated on the other remaining node).

          What is now happening is that out of 5 tree cache elements (nodes) removed, only 3 of these are actually deleted from the cache. I display each cache node as it is deleted so I know how many and which ones are targeted. 5 nodes are displayed but afterwards, two of those nodes are still in the cache!

          I use Pojo cache and these 5 nodes are part of the same Java Set so I cannot fathom why some of the nodes would be deleted and some would remain. I iterate through the set and look for matches to my search criteria. When I find a match, I remove the object from the Set. Please tell me how this can fail to work. Thanks.