2 Replies Latest reply on Jul 30, 2007 7:52 AM by manik

    JBoss Cache 1.4.1 SP3 | NodeLockingScheme

    sg283
    sg283 Jul 26, 2007 7:54 AM

      We are using JBoss AS 4.0.3 SP1 and JBoss Cache 1.4.1 SP3 and MySQL Server 50.18. We are using PESSIMISTIC NodeLockingScheme currently.
      We are getting various cache exceptions:-
      1- DataVersioning
      2- Duplicate Key
      3- ReadWriteLock

      Because of these errors, our production system gets down.
      Here is the exception stack trace:-

      ReadWriteLock Exception

      26 Jul 07 08:05:36, INFO org.jboss.cache.interceptors.TxInterceptor:invoke:161 There was a problem handling this request
      java.lang.InterruptedException
      at EDU.oswego.cs.dl.util.concurrent.NullSync.attempt(NullSync.java:38)
      at org.jboss.cache.lock.IdentityLock.acquireReadLock(IdentityLock.java:248)
      at org.jboss.cache.Node.acquireReadLock(Node.java:417)
      at org.jboss.cache.Node.acquire(Node.java:384)
      at org.jboss.cache.interceptors.PessimisticLockInterceptor.lock(PessimisticLockInterceptor.java:231)
      at org.jboss.cache.interceptors.PessimisticLockInterceptor.invoke(PessimisticLockInterceptor.java:160)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:186)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:197)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:345)
      at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:156)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:157)
      at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5520)
      at org.jboss.cache.TreeCache.put(TreeCache.java:3678)
      at com.tg.cache.manager.FareCacheProviderImpl.addToCache(FareCacheProviderImpl.java:104)
      at com.tg.ws.abacus.AbacusFareService.putResponseInCache(AbacusFareService.java:615)
      at com.tg.ws.abacus.AbacusFareService_1_5.getFaresFromCacheOrWSForRequest(AbacusFareService_1_5.java:189)
      at com.tg.ws.abacus.FarePoolHandler.execute(FarePoolHandler.java:29)
      at com.tg.ws.abacus.FarePoolHandler.execute(FarePoolHandler.java:15)
      at com.tg.parallel.concurrent.JavaConcurrentPoolExecutor$CallableAdapter.call(JavaConcurrentPoolExecutor.java:93)
      at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
      at java.util.concurrent.FutureTask.run(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at java.lang.Thread.run(Unknown Source)


      Duplicate Key Exception

      26 Jul 07 08:38:23, ERROR org.jboss.cache.loader.JDBCCacheLoader:insertNode:981 Failed to insert node: Duplicate entry '/fare/JDH/UDR/3' for key 1
      com.mysql.jdbc.exceptions.MySQLIntegrityConstraintViolationException: Duplicate entry '/fare/JDH/UDR/3' for key 1
      at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:931)
      at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2870)
      at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1573)
      at com.mysql.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:1160)
      at com.mysql.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:685)
      at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1400)
      at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1314)
      at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1299)
      at org.jboss.cache.loader.JDBCCacheLoader.insertNode(JDBCCacheLoader.java:969)
      at org.jboss.cache.loader.JDBCCacheLoader.put(JDBCCacheLoader.java:321)
      at org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:216)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:197)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:345)
      at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:156)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:157)
      at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5520)
      at org.jboss.cache.TreeCache.put(TreeCache.java:3678)
      at com.tg.cache.manager.FareCacheProviderImpl.addToCache(FareCacheProviderImpl.java:104)
      at com.tg.ws.abacus.AbacusFareService.putResponseInCache(AbacusFareService.java:615)
      at com.tg.ws.abacus.AbacusFareService_1_5.getFaresFromCacheOrWSForRequest(AbacusFareService_1_5.java:187)
      at com.tg.ws.abacus.FarePoolHandler.execute(FarePoolHandler.java:29)
      at com.tg.ws.abacus.FarePoolHandler.execute(FarePoolHandler.java:15)
      at com.tg.parallel.concurrent.JavaConcurrentPoolExecutor$CallableAdapter.call(JavaConcurrentPoolExecutor.java:93)
      at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
      at java.util.concurrent.FutureTask.run(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at java.lang.Thread.run(Unknown Source)

      Data Versioning Exception
      26 Jul 07 05:36:30, ERROR org.jboss.cache.transaction.DummyTransaction:notifyBeforeCompletion:251 beforeCompletion() failed for tx=org.jboss.cache.transaction.DummyTransaction@b2e747, handlers=[TxInterceptor.LocalSynchronizationHandler(gtx=GlobalTransaction::202712, tx=org.jboss.cache.transaction.DummyTransaction@b2e747)]
      java.lang.RuntimeException:
      at org.jboss.cache.interceptors.TxInterceptor$LocalSynchronizationHandler.beforeCompletion(TxInterceptor.java:1091)
      at org.jboss.cache.interceptors.OrderedSynchronizationHandler.beforeCompletion(OrderedSynchronizationHandler.java:75)
      at org.jboss.cache.transaction.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:247)
      at org.jboss.cache.transaction.DummyTransaction.commit(DummyTransaction.java:54)
      at org.jboss.cache.transaction.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:61)
      at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:349)
      at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:156)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:157)
      at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5520)
      at org.jboss.cache.TreeCache.put(TreeCache.java:3678)
      at com.tg.cache.manager.FareCacheProviderImpl.addToCache(FareCacheProviderImpl.java:104)
      at com.tg.ws.abacus.AbacusFareService.putResponseInCache(AbacusFareService.java:615)
      at com.tg.ws.abacus.AbacusFareService_1_5.getFaresFromCacheOrWSForRequest(AbacusFareService_1_5.java:189)
      at com.tg.ws.abacus.FarePoolHandler.execute(FarePoolHandler.java:29)
      at com.tg.ws.abacus.FarePoolHandler.execute(FarePoolHandler.java:15)
      at com.tg.parallel.concurrent.JavaConcurrentPoolExecutor$CallableAdapter.call(JavaConcurrentPoolExecutor.java:93)
      at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
      at java.util.concurrent.FutureTask.run(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at java.lang.Thread.run(Unknown Source)
      Caused by: org.jboss.cache.CacheException: DataNode [/fare/DEL/MAA/8/15/07 12:00 AM] version Ver=3 is newer than workspace node Ver=2
      at org.jboss.cache.interceptors.OptimisticValidatorInterceptor.simpleValidate(OptimisticValidatorInterceptor.java:127)
      at org.jboss.cache.interceptors.OptimisticValidatorInterceptor.validateNodes(OptimisticValidatorInterceptor.java:101)
      at org.jboss.cache.interceptors.OptimisticValidatorInterceptor.invoke(OptimisticValidatorInterceptor.java:66)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.OptimisticLockingInterceptor.invoke(OptimisticLockingInterceptor.java:95)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:135)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:197)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.TxInterceptor.runPreparePhase(TxInterceptor.java:804)
      at org.jboss.cache.interceptors.TxInterceptor$LocalSynchronizationHandler.beforeCompletion(TxInterceptor.java:1069)
      ... 21 more

      • 1764 Views
      • Tags:
        • 1. JBoss Cache 1.4.1 SP3 |Problem with OPTIMISTIC NodeLockingSc
          sg283
          sg283 Jul 27, 2007 2:16 AM (in response to sg283)

          We are using JBoss AS 4.0.3 SP1 and JBoss Cache 1.4.1 SP3 and MySQL Server 5.0.18. We are using OPTIMISTIC NodeLockingScheme currently.
          We are getting various cache exceptions:-
          1- DataVersioning
          2- Duplicate Key
          3- ReadWriteLock

          Because of these errors, our production system gets down.
          Here is the exception stack trace:-

          ReadWriteLock Exception

          26 Jul 07 08:05:36, INFO org.jboss.cache.interceptors.TxInterceptor:invoke:161 There was a problem handling this request
          java.lang.InterruptedException
          at EDU.oswego.cs.dl.util.concurrent.NullSync.attempt(NullSync.java:38)
          at org.jboss.cache.lock.IdentityLock.acquireReadLock(IdentityLock.java:248)
          at org.jboss.cache.Node.acquireReadLock(Node.java:417)
          at org.jboss.cache.Node.acquire(Node.java:384)
          at org.jboss.cache.interceptors.PessimisticLockInterceptor.lock(PessimisticLockInterceptor.java:231)
          at org.jboss.cache.interceptors.PessimisticLockInterceptor.invoke(PessimisticLockInterceptor.java:160)
          at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
          at org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:186)
          at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
          at org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:197)
          at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
          at org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
          at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
          at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:345)
          at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:156)
          at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
          at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:157)
          at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5520)
          at org.jboss.cache.TreeCache.put(TreeCache.java:3678)
          at com.tg.cache.manager.FareCacheProviderImpl.addToCache(FareCacheProviderImpl.java:104)
          at com.tg.ws.abacus.AbacusFareService.putResponseInCache(AbacusFareService.java:615)
          at com.tg.ws.abacus.AbacusFareService_1_5.getFaresFromCacheOrWSForRequest(AbacusFareService_1_5.java:189)
          at com.tg.ws.abacus.FarePoolHandler.execute(FarePoolHandler.java:29)
          at com.tg.ws.abacus.FarePoolHandler.execute(FarePoolHandler.java:15)
          at com.tg.parallel.concurrent.JavaConcurrentPoolExecutor$CallableAdapter.call(JavaConcurrentPoolExecutor.java:93)
          at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at java.lang.Thread.run(Unknown Source)


          Duplicate Key Exception

          26 Jul 07 08:38:23, ERROR org.jboss.cache.loader.JDBCCacheLoader:insertNode:981 Failed to insert node: Duplicate entry '/fare/JDH/UDR/3' for key 1
          com.mysql.jdbc.exceptions.MySQLIntegrityConstraintViolationException: Duplicate entry '/fare/JDH/UDR/3' for key 1
          at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:931)
          at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2870)
          at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1573)
          at com.mysql.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:1160)
          at com.mysql.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:685)
          at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1400)
          at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1314)
          at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1299)
          at org.jboss.cache.loader.JDBCCacheLoader.insertNode(JDBCCacheLoader.java:969)
          at org.jboss.cache.loader.JDBCCacheLoader.put(JDBCCacheLoader.java:321)
          at org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:216)
          at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
          at org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:197)
          at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
          at org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
          at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
          at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:345)
          at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:156)
          at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
          at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:157)
          at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5520)
          at org.jboss.cache.TreeCache.put(TreeCache.java:3678)
          at com.tg.cache.manager.FareCacheProviderImpl.addToCache(FareCacheProviderImpl.java:104)
          at com.tg.ws.abacus.AbacusFareService.putResponseInCache(AbacusFareService.java:615)
          at com.tg.ws.abacus.AbacusFareService_1_5.getFaresFromCacheOrWSForRequest(AbacusFareService_1_5.java:187)
          at com.tg.ws.abacus.FarePoolHandler.execute(FarePoolHandler.java:29)
          at com.tg.ws.abacus.FarePoolHandler.execute(FarePoolHandler.java:15)
          at com.tg.parallel.concurrent.JavaConcurrentPoolExecutor$CallableAdapter.call(JavaConcurrentPoolExecutor.java:93)
          at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at java.lang.Thread.run(Unknown Source)

          Data Versioning Exception
          26 Jul 07 05:36:30, ERROR org.jboss.cache.transaction.DummyTransaction:notifyBeforeCompletion:251 beforeCompletion() failed for tx=org.jboss.cache.transaction.DummyTransaction@b2e747, handlers=[TxInterceptor.LocalSynchronizationHandler(gtx=GlobalTransaction::202712, tx=org.jboss.cache.transaction.DummyTransaction@b2e747)]
          java.lang.RuntimeException:
          at org.jboss.cache.interceptors.TxInterceptor$LocalSynchronizationHandler.beforeCompletion(TxInterceptor.java:1091)
          at org.jboss.cache.interceptors.OrderedSynchronizationHandler.beforeCompletion(OrderedSynchronizationHandler.java:75)
          at org.jboss.cache.transaction.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:247)
          at org.jboss.cache.transaction.DummyTransaction.commit(DummyTransaction.java:54)
          at org.jboss.cache.transaction.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:61)
          at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:349)
          at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:156)
          at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
          at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:157)
          at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5520)
          at org.jboss.cache.TreeCache.put(TreeCache.java:3678)
          at com.tg.cache.manager.FareCacheProviderImpl.addToCache(FareCacheProviderImpl.java:104)
          at com.tg.ws.abacus.AbacusFareService.putResponseInCache(AbacusFareService.java:615)
          at com.tg.ws.abacus.AbacusFareService_1_5.getFaresFromCacheOrWSForRequest(AbacusFareService_1_5.java:189)
          at com.tg.ws.abacus.FarePoolHandler.execute(FarePoolHandler.java:29)
          at com.tg.ws.abacus.FarePoolHandler.execute(FarePoolHandler.java:15)
          at com.tg.parallel.concurrent.JavaConcurrentPoolExecutor$CallableAdapter.call(JavaConcurrentPoolExecutor.java:93)
          at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at java.lang.Thread.run(Unknown Source)
          Caused by: org.jboss.cache.CacheException: DataNode [/fare/DEL/MAA/8/15/07 12:00 AM] version Ver=3 is newer than workspace node Ver=2
          at org.jboss.cache.interceptors.OptimisticValidatorInterceptor.simpleValidate(OptimisticValidatorInterceptor.java:127)
          at org.jboss.cache.interceptors.OptimisticValidatorInterceptor.validateNodes(OptimisticValidatorInterceptor.java:101)
          at org.jboss.cache.interceptors.OptimisticValidatorInterceptor.invoke(OptimisticValidatorInterceptor.java:66)
          at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
          at org.jboss.cache.interceptors.OptimisticLockingInterceptor.invoke(OptimisticLockingInterceptor.java:95)
          at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
          at org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:135)
          at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
          at org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:197)
          at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
          at org.jboss.cache.interceptors.TxInterceptor.runPreparePhase(TxInterceptor.java:804)
          at org.jboss.cache.interceptors.TxInterceptor$LocalSynchronizationHandler.beforeCompletion(TxInterceptor.java:1069)
          ... 21 more

            • Actions
          • 2. Re: JBoss Cache 1.4.1 SP3 | NodeLockingScheme
            manik
            manik Jul 30, 2007 7:52 AM (in response to sg283)

            Some of your data versioning exceptions look like expected optimistic collisions.

            All the same, I would try 1.4.1.SP4 as it does contain quite a few enhancements to the optimistic validation codebase.

              • Actions