0 Replies Latest reply on Sep 15, 2015 12:00 PM by stephen lau

    WriteSkewException after cluster-wide rebalance

    stephen lau Newbie

      Hi,

       

      I am getting WriteSkewException after cluster-wide rebalance.   How can I avoid or handle this error?    The exception and the cache configuration are shown below:

       

      2015/09/14-14:09:06,643 INFO  [org.infinispan.CLUSTER] (remote-thread-0,Prod-Dev-JCM-2) ISPN000310: Starting cluster-wide rebalance for cache AcanoMap-activeBank-v1_0, topology CacheTopology{id=1, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 60, owners = (1)[Prod-Dev-JCM-2-16835: 60]}, pendingCH=ReplicatedConsistentHash{ns = 60, owners = (2)[Prod-Dev-JCM-2-16835: 30, Prod-Dev-JCM-1-50974: 30]}, unionCH=null, actualMembers=[Prod-Dev-JCM-2-16835, Prod-Dev-JCM-1-50974]}

      2015/09/14-14:09:06,666 INFO  [org.infinispan.CLUSTER] (remote-thread-1,Prod-Dev-JCM-2) ISPN000336: Finished cluster-wide rebalance for cache AcanoMap-activeBank-v1_0, topology id = 1

      2015/09/14-14:09:07,003 ERROR [org.infinispan.remoting.InboundInvocationHandlerImpl] (remote-thread-1,Prod-Dev-JCM-2) ISPN000260: Exception executing command: org.infinispan.transaction.WriteSkewException: Write skew detected on key ACANO-c0505f85-4a94-495e-ba13-a8d8add060bd for transaction null

              at org.infinispan.transaction.WriteSkewHelper.performWriteSkewCheckAndReturnNewVersions(WriteSkewHelper.java:53) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.clusteredCreateNewVersionsAndCheckForWriteSkews(ClusteringDependentLogic.java:148) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.createNewVersionsAndCheckForWriteSkews(ClusteringDependentLogic.java:89) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.VersionedEntryWrappingInterceptor.visitPrepareCommand(VersionedEntryWrappingInterceptor.java:53) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:78) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:87) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:36) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:132) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:119) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:103) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:39) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:244) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:171) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:99) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:103) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:103) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:104) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:103) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.commands.tx.PrepareCommand.perform(PrepareCommand.java:115) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:92) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:47) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:184) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_80]

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_80]

              at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_80]

       

       

      2015/09/14-14:09:23,698 ERROR [org.infinispan.remoting.InboundInvocationHandlerImpl] (remote-thread-1,Prod-Dev-JCM-2) ISPN000260: Exception executing command: org.infinispan.transaction.WriteSkewException: Write skew detected on key ACANO-c0505f85-4a94-495e-ba13-a8d8add060bd for transaction null

              at org.infinispan.transaction.WriteSkewHelper.performWriteSkewCheckAndReturnNewVersions(WriteSkewHelper.java:53) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.clusteredCreateNewVersionsAndCheckForWriteSkews(ClusteringDependentLogic.java:148) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.createNewVersionsAndCheckForWriteSkews(ClusteringDependentLogic.java:89) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.VersionedEntryWrappingInterceptor.visitPrepareCommand(VersionedEntryWrappingInterceptor.java:53) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:78) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:87) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:36) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:132) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:119) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:103) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:39) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:244) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:171) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:99) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:103) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:103) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:104) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:103) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.commands.tx.PrepareCommand.perform(PrepareCommand.java:115) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:92) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:47) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:184) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_80]

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_80]

              at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_80]

       

       

      Cache configuration:

        <default>

          <clustering mode="replication">

            <sync/>

          </clustering>

          <transaction

             transactionManagerLookupClass="org.infinispan.transaction.lookup.GenericTransactionManagerLookup"

             transactionMode="TRANSACTIONAL"

             lockingMode="OPTIMISTIC"

             />

        <locking

             isolationLevel="REPEATABLE_READ"

             writeSkewCheck="true"

         />

        <versioning

        enabled="true"

        versioningScheme="SIMPLE"

        />

        <persistence passivation="false">

        <singleFile

             shared="false"

             preload="true"

             fetchPersistentState="false"

             ignoreModifications="false"

             purgeOnStartup="true"

             location="${jboss.server.data.dir}/datagrid">

        <async

        enabled="true"

        flushLockTimeout="10"

        modificationQueueSize="1024"

        threadPoolSize="5" />

        <singleton

             enabled="false" />

        </singleFile>

        </persistence>

        </default>