Infinispan cluster become unstable randomly
hettimora1979 Jun 24, 2016 3:45 AMWe are using the UDP base infinispan cluster with two node. But randomly both node stop responding with the following error.
Our cache configuration detail
<default>
<eviction strategy="NONE"/>
<transaction
transactionMode="NON_TRANSACTIONAL"
lockingMode="OPTIMISTIC"
transactionManagerLookupClass="org.infinispan.transaction.lookup.GenericTransactionManagerLookup"
syncRollbackPhase="false"
syncCommitPhase="false"
useEagerLocking="false"
/>
<locking concurrencyLevel="1000" isolationLevel="READ_COMMITTED" lockAcquisitionTimeout="5000" useLockStriping="false"/>
<!--<clustering mode="local">
<async/>
</clustering>-->
<clustering mode="repl">
<async/>
</clustering>
<!-- <locking useLockStriping ="false" />-->
<indexing enabled="false" indexLocalOnly="false">
<properties>
<property name="hibernate.search.default.directory_provider" value="ram"/>
<property name="hibernate.search.lucene_version" value="LUCENE_36"/>
<property name="hibernate.search.default.exclusive_index_use" value="false"/>
<property name="hibernate.search.Books.locking_strategy" value="org.apache.lucene.store.NoLockFactory"/>
</properties>
</indexing>
</default>
Error which cause
0m [31m2016-06-20 10:16:35,719 ERROR (InvocationContextInterceptor.java:148) (Incoming-2,MFSNEWGENUAT-43882)() ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [10 seconds] on key [DFM-DNIR] for requestor [Thread[Incoming-2,MFSNEWGENUAT-43882,5,Thread Pools]]! Lock held by [Thread[EJB default - 6,5,EJB default]]
at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:213) [infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLockNoCheck(LockManagerImpl.java:196) [infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockKey(AbstractLockingInterceptor.java:114) [infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.lockKey(NonTransactionalLockingInterceptor.java:178) [infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitPutKeyValueCommand(NonTransactionalLockingInterceptor.java:72) [infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80) [infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) [infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62) [infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80) [infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) [infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62) [infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80) [infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216) [infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194) [infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136) [infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80) [infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) [infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) [infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62) [infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:80) [infinispan-core-5.2.7.Final.jar:5.2.7.Final]