WriteSkewException after cluster-wide rebalance
Hi,
I am getting WriteSkewException after cluster-wide rebalance. How can I avoid or handle this error? The exception and the cache configuration are shown below:
2015/09/14-14:09:06,643 INFO [org.infinispan.CLUSTER] (remote-thread-0,Prod-Dev-JCM-2) ISPN000310: Starting cluster-wide rebalance for cache AcanoMap-activeBank-v1_0, topology CacheTopology{id=1, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 60, owners = (1)[Prod-Dev-JCM-2-16835: 60]}, pendingCH=ReplicatedConsistentHash{ns = 60, owners = (2)[Prod-Dev-JCM-2-16835: 30, Prod-Dev-JCM-1-50974: 30]}, unionCH=null, actualMembers=[Prod-Dev-JCM-2-16835, Prod-Dev-JCM-1-50974]}
2015/09/14-14:09:06,666 INFO [org.infinispan.CLUSTER] (remote-thread-1,Prod-Dev-JCM-2) ISPN000336: Finished cluster-wide rebalance for cache AcanoMap-activeBank-v1_0, topology id = 1
2015/09/14-14:09:07,003 ERROR [org.infinispan.remoting.InboundInvocationHandlerImpl] (remote-thread-1,Prod-Dev-JCM-2) ISPN000260: Exception executing command: org.infinispan.transaction.WriteSkewException: Write skew detected on key ACANO-c0505f85-4a94-495e-ba13-a8d8add060bd for transaction null
at org.infinispan.transaction.WriteSkewHelper.performWriteSkewCheckAndReturnNewVersions(WriteSkewHelper.java:53) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.clusteredCreateNewVersionsAndCheckForWriteSkews(ClusteringDependentLogic.java:148) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.createNewVersionsAndCheckForWriteSkews(ClusteringDependentLogic.java:89) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.VersionedEntryWrappingInterceptor.visitPrepareCommand(VersionedEntryWrappingInterceptor.java:53) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:78) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:87) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:36) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:132) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:119) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:103) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:39) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:244) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:171) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:99) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:103) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:103) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:104) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:103) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.commands.tx.PrepareCommand.perform(PrepareCommand.java:115) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:92) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:47) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:184) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_80]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_80]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_80]
2015/09/14-14:09:23,698 ERROR [org.infinispan.remoting.InboundInvocationHandlerImpl] (remote-thread-1,Prod-Dev-JCM-2) ISPN000260: Exception executing command: org.infinispan.transaction.WriteSkewException: Write skew detected on key ACANO-c0505f85-4a94-495e-ba13-a8d8add060bd for transaction null
at org.infinispan.transaction.WriteSkewHelper.performWriteSkewCheckAndReturnNewVersions(WriteSkewHelper.java:53) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.clusteredCreateNewVersionsAndCheckForWriteSkews(ClusteringDependentLogic.java:148) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.createNewVersionsAndCheckForWriteSkews(ClusteringDependentLogic.java:89) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.VersionedEntryWrappingInterceptor.visitPrepareCommand(VersionedEntryWrappingInterceptor.java:53) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:78) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:87) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:36) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:132) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:119) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:103) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:39) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:244) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:171) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:99) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:103) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:103) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:104) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:103) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.commands.tx.PrepareCommand.perform(PrepareCommand.java:115) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:92) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:47) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:184) [infinispan-core-6.2.1.Final-redhat-2.jar:6.2.1.Final-redhat-2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_80]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_80]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_80]
Cache configuration:
<default>
<clustering mode="replication">
<sync/>
</clustering>
<transaction
transactionManagerLookupClass="org.infinispan.transaction.lookup.GenericTransactionManagerLookup"
transactionMode="TRANSACTIONAL"
lockingMode="OPTIMISTIC"
/>
<locking
isolationLevel="REPEATABLE_READ"
writeSkewCheck="true"
/>
<versioning
enabled="true"
versioningScheme="SIMPLE"
/>
<persistence passivation="false">
<singleFile
shared="false"
preload="true"
fetchPersistentState="false"
ignoreModifications="false"
purgeOnStartup="true"
location="${jboss.server.data.dir}/datagrid">
<async
enabled="true"
flushLockTimeout="10"
modificationQueueSize="1024"
threadPoolSize="5" />
<singleton
enabled="false" />
</singleFile>
</persistence>
</default>