2 Replies Latest reply on Jul 30, 2007 7:52 AM by Manik Surtani

    JBoss Cache 1.4.1 SP3 | NodeLockingScheme

    Saurabh Gupta Newbie

      We are using JBoss AS 4.0.3 SP1 and JBoss Cache 1.4.1 SP3 and MySQL Server 50.18. We are using PESSIMISTIC NodeLockingScheme currently.
      We are getting various cache exceptions:-
      1- DataVersioning
      2- Duplicate Key
      3- ReadWriteLock

      Because of these errors, our production system gets down.
      Here is the exception stack trace:-

      ReadWriteLock Exception

      26 Jul 07 08:05:36, INFO org.jboss.cache.interceptors.TxInterceptor:invoke:161 There was a problem handling this request
      java.lang.InterruptedException
      at EDU.oswego.cs.dl.util.concurrent.NullSync.attempt(NullSync.java:38)
      at org.jboss.cache.lock.IdentityLock.acquireReadLock(IdentityLock.java:248)
      at org.jboss.cache.Node.acquireReadLock(Node.java:417)
      at org.jboss.cache.Node.acquire(Node.java:384)
      at org.jboss.cache.interceptors.PessimisticLockInterceptor.lock(PessimisticLockInterceptor.java:231)
      at org.jboss.cache.interceptors.PessimisticLockInterceptor.invoke(PessimisticLockInterceptor.java:160)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:186)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:197)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:345)
      at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:156)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:157)
      at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5520)
      at org.jboss.cache.TreeCache.put(TreeCache.java:3678)
      at com.tg.cache.manager.FareCacheProviderImpl.addToCache(FareCacheProviderImpl.java:104)
      at com.tg.ws.abacus.AbacusFareService.putResponseInCache(AbacusFareService.java:615)
      at com.tg.ws.abacus.AbacusFareService_1_5.getFaresFromCacheOrWSForRequest(AbacusFareService_1_5.java:189)
      at com.tg.ws.abacus.FarePoolHandler.execute(FarePoolHandler.java:29)
      at com.tg.ws.abacus.FarePoolHandler.execute(FarePoolHandler.java:15)
      at com.tg.parallel.concurrent.JavaConcurrentPoolExecutor$CallableAdapter.call(JavaConcurrentPoolExecutor.java:93)
      at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
      at java.util.concurrent.FutureTask.run(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at java.lang.Thread.run(Unknown Source)


      Duplicate Key Exception

      26 Jul 07 08:38:23, ERROR org.jboss.cache.loader.JDBCCacheLoader:insertNode:981 Failed to insert node: Duplicate entry '/fare/JDH/UDR/3' for key 1
      com.mysql.jdbc.exceptions.MySQLIntegrityConstraintViolationException: Duplicate entry '/fare/JDH/UDR/3' for key 1
      at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:931)
      at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2870)
      at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1573)
      at com.mysql.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:1160)
      at com.mysql.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:685)
      at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1400)
      at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1314)
      at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1299)
      at org.jboss.cache.loader.JDBCCacheLoader.insertNode(JDBCCacheLoader.java:969)
      at org.jboss.cache.loader.JDBCCacheLoader.put(JDBCCacheLoader.java:321)
      at org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:216)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:197)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:345)
      at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:156)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:157)
      at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5520)
      at org.jboss.cache.TreeCache.put(TreeCache.java:3678)
      at com.tg.cache.manager.FareCacheProviderImpl.addToCache(FareCacheProviderImpl.java:104)
      at com.tg.ws.abacus.AbacusFareService.putResponseInCache(AbacusFareService.java:615)
      at com.tg.ws.abacus.AbacusFareService_1_5.getFaresFromCacheOrWSForRequest(AbacusFareService_1_5.java:187)
      at com.tg.ws.abacus.FarePoolHandler.execute(FarePoolHandler.java:29)
      at com.tg.ws.abacus.FarePoolHandler.execute(FarePoolHandler.java:15)
      at com.tg.parallel.concurrent.JavaConcurrentPoolExecutor$CallableAdapter.call(JavaConcurrentPoolExecutor.java:93)
      at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
      at java.util.concurrent.FutureTask.run(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at java.lang.Thread.run(Unknown Source)

      Data Versioning Exception
      26 Jul 07 05:36:30, ERROR org.jboss.cache.transaction.DummyTransaction:notifyBeforeCompletion:251 beforeCompletion() failed for tx=org.jboss.cache.transaction.DummyTransaction@b2e747, handlers=[TxInterceptor.LocalSynchronizationHandler(gtx=GlobalTransaction::202712, tx=org.jboss.cache.transaction.DummyTransaction@b2e747)]
      java.lang.RuntimeException:
      at org.jboss.cache.interceptors.TxInterceptor$LocalSynchronizationHandler.beforeCompletion(TxInterceptor.java:1091)
      at org.jboss.cache.interceptors.OrderedSynchronizationHandler.beforeCompletion(OrderedSynchronizationHandler.java:75)
      at org.jboss.cache.transaction.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:247)
      at org.jboss.cache.transaction.DummyTransaction.commit(DummyTransaction.java:54)
      at org.jboss.cache.transaction.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:61)
      at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:349)
      at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:156)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:157)
      at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5520)
      at org.jboss.cache.TreeCache.put(TreeCache.java:3678)
      at com.tg.cache.manager.FareCacheProviderImpl.addToCache(FareCacheProviderImpl.java:104)
      at com.tg.ws.abacus.AbacusFareService.putResponseInCache(AbacusFareService.java:615)
      at com.tg.ws.abacus.AbacusFareService_1_5.getFaresFromCacheOrWSForRequest(AbacusFareService_1_5.java:189)
      at com.tg.ws.abacus.FarePoolHandler.execute(FarePoolHandler.java:29)
      at com.tg.ws.abacus.FarePoolHandler.execute(FarePoolHandler.java:15)
      at com.tg.parallel.concurrent.JavaConcurrentPoolExecutor$CallableAdapter.call(JavaConcurrentPoolExecutor.java:93)
      at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
      at java.util.concurrent.FutureTask.run(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at java.lang.Thread.run(Unknown Source)
      Caused by: org.jboss.cache.CacheException: DataNode [/fare/DEL/MAA/8/15/07 12:00 AM] version Ver=3 is newer than workspace node Ver=2
      at org.jboss.cache.interceptors.OptimisticValidatorInterceptor.simpleValidate(OptimisticValidatorInterceptor.java:127)
      at org.jboss.cache.interceptors.OptimisticValidatorInterceptor.validateNodes(OptimisticValidatorInterceptor.java:101)
      at org.jboss.cache.interceptors.OptimisticValidatorInterceptor.invoke(OptimisticValidatorInterceptor.java:66)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.OptimisticLockingInterceptor.invoke(OptimisticLockingInterceptor.java:95)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:135)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:197)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.TxInterceptor.runPreparePhase(TxInterceptor.java:804)
      at org.jboss.cache.interceptors.TxInterceptor$LocalSynchronizationHandler.beforeCompletion(TxInterceptor.java:1069)
      ... 21 more

        • 1. JBoss Cache 1.4.1 SP3 |Problem with OPTIMISTIC NodeLockingSc
          Saurabh Gupta Newbie

          We are using JBoss AS 4.0.3 SP1 and JBoss Cache 1.4.1 SP3 and MySQL Server 5.0.18. We are using OPTIMISTIC NodeLockingScheme currently.
          We are getting various cache exceptions:-
          1- DataVersioning
          2- Duplicate Key
          3- ReadWriteLock

          Because of these errors, our production system gets down.
          Here is the exception stack trace:-

          ReadWriteLock Exception

          26 Jul 07 08:05:36, INFO org.jboss.cache.interceptors.TxInterceptor:invoke:161 There was a problem handling this request
          java.lang.InterruptedException
          at EDU.oswego.cs.dl.util.concurrent.NullSync.attempt(NullSync.java:38)
          at org.jboss.cache.lock.IdentityLock.acquireReadLock(IdentityLock.java:248)
          at org.jboss.cache.Node.acquireReadLock(Node.java:417)
          at org.jboss.cache.Node.acquire(Node.java:384)
          at org.jboss.cache.interceptors.PessimisticLockInterceptor.lock(PessimisticLockInterceptor.java:231)
          at org.jboss.cache.interceptors.PessimisticLockInterceptor.invoke(PessimisticLockInterceptor.java:160)
          at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
          at org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:186)
          at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
          at org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:197)
          at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
          at org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
          at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
          at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:345)
          at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:156)
          at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
          at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:157)
          at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5520)
          at org.jboss.cache.TreeCache.put(TreeCache.java:3678)
          at com.tg.cache.manager.FareCacheProviderImpl.addToCache(FareCacheProviderImpl.java:104)
          at com.tg.ws.abacus.AbacusFareService.putResponseInCache(AbacusFareService.java:615)
          at com.tg.ws.abacus.AbacusFareService_1_5.getFaresFromCacheOrWSForRequest(AbacusFareService_1_5.java:189)
          at com.tg.ws.abacus.FarePoolHandler.execute(FarePoolHandler.java:29)
          at com.tg.ws.abacus.FarePoolHandler.execute(FarePoolHandler.java:15)
          at com.tg.parallel.concurrent.JavaConcurrentPoolExecutor$CallableAdapter.call(JavaConcurrentPoolExecutor.java:93)
          at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at java.lang.Thread.run(Unknown Source)


          Duplicate Key Exception

          26 Jul 07 08:38:23, ERROR org.jboss.cache.loader.JDBCCacheLoader:insertNode:981 Failed to insert node: Duplicate entry '/fare/JDH/UDR/3' for key 1
          com.mysql.jdbc.exceptions.MySQLIntegrityConstraintViolationException: Duplicate entry '/fare/JDH/UDR/3' for key 1
          at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:931)
          at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2870)
          at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1573)
          at com.mysql.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:1160)
          at com.mysql.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:685)
          at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1400)
          at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1314)
          at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1299)
          at org.jboss.cache.loader.JDBCCacheLoader.insertNode(JDBCCacheLoader.java:969)
          at org.jboss.cache.loader.JDBCCacheLoader.put(JDBCCacheLoader.java:321)
          at org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:216)
          at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
          at org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:197)
          at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
          at org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
          at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
          at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:345)
          at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:156)
          at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
          at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:157)
          at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5520)
          at org.jboss.cache.TreeCache.put(TreeCache.java:3678)
          at com.tg.cache.manager.FareCacheProviderImpl.addToCache(FareCacheProviderImpl.java:104)
          at com.tg.ws.abacus.AbacusFareService.putResponseInCache(AbacusFareService.java:615)
          at com.tg.ws.abacus.AbacusFareService_1_5.getFaresFromCacheOrWSForRequest(AbacusFareService_1_5.java:187)
          at com.tg.ws.abacus.FarePoolHandler.execute(FarePoolHandler.java:29)
          at com.tg.ws.abacus.FarePoolHandler.execute(FarePoolHandler.java:15)
          at com.tg.parallel.concurrent.JavaConcurrentPoolExecutor$CallableAdapter.call(JavaConcurrentPoolExecutor.java:93)
          at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at java.lang.Thread.run(Unknown Source)

          Data Versioning Exception
          26 Jul 07 05:36:30, ERROR org.jboss.cache.transaction.DummyTransaction:notifyBeforeCompletion:251 beforeCompletion() failed for tx=org.jboss.cache.transaction.DummyTransaction@b2e747, handlers=[TxInterceptor.LocalSynchronizationHandler(gtx=GlobalTransaction::202712, tx=org.jboss.cache.transaction.DummyTransaction@b2e747)]
          java.lang.RuntimeException:
          at org.jboss.cache.interceptors.TxInterceptor$LocalSynchronizationHandler.beforeCompletion(TxInterceptor.java:1091)
          at org.jboss.cache.interceptors.OrderedSynchronizationHandler.beforeCompletion(OrderedSynchronizationHandler.java:75)
          at org.jboss.cache.transaction.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:247)
          at org.jboss.cache.transaction.DummyTransaction.commit(DummyTransaction.java:54)
          at org.jboss.cache.transaction.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:61)
          at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:349)
          at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:156)
          at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
          at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:157)
          at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5520)
          at org.jboss.cache.TreeCache.put(TreeCache.java:3678)
          at com.tg.cache.manager.FareCacheProviderImpl.addToCache(FareCacheProviderImpl.java:104)
          at com.tg.ws.abacus.AbacusFareService.putResponseInCache(AbacusFareService.java:615)
          at com.tg.ws.abacus.AbacusFareService_1_5.getFaresFromCacheOrWSForRequest(AbacusFareService_1_5.java:189)
          at com.tg.ws.abacus.FarePoolHandler.execute(FarePoolHandler.java:29)
          at com.tg.ws.abacus.FarePoolHandler.execute(FarePoolHandler.java:15)
          at com.tg.parallel.concurrent.JavaConcurrentPoolExecutor$CallableAdapter.call(JavaConcurrentPoolExecutor.java:93)
          at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at java.lang.Thread.run(Unknown Source)
          Caused by: org.jboss.cache.CacheException: DataNode [/fare/DEL/MAA/8/15/07 12:00 AM] version Ver=3 is newer than workspace node Ver=2
          at org.jboss.cache.interceptors.OptimisticValidatorInterceptor.simpleValidate(OptimisticValidatorInterceptor.java:127)
          at org.jboss.cache.interceptors.OptimisticValidatorInterceptor.validateNodes(OptimisticValidatorInterceptor.java:101)
          at org.jboss.cache.interceptors.OptimisticValidatorInterceptor.invoke(OptimisticValidatorInterceptor.java:66)
          at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
          at org.jboss.cache.interceptors.OptimisticLockingInterceptor.invoke(OptimisticLockingInterceptor.java:95)
          at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
          at org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:135)
          at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
          at org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:197)
          at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
          at org.jboss.cache.interceptors.TxInterceptor.runPreparePhase(TxInterceptor.java:804)
          at org.jboss.cache.interceptors.TxInterceptor$LocalSynchronizationHandler.beforeCompletion(TxInterceptor.java:1069)
          ... 21 more

          • 2. Re: JBoss Cache 1.4.1 SP3 | NodeLockingScheme
            Manik Surtani Master

            Some of your data versioning exceptions look like expected optimistic collisions.

            All the same, I would try 1.4.1.SP4 as it does contain quite a few enhancements to the optimistic validation codebase.