1 Reply Latest reply on Apr 28, 2016 9:47 AM by Galder Zamarreño

    Stuck thread while DecoratedCache.put invoked

    tomas11 Novice

      Hi all

       

      We have noticed following behavior. We have embedded Infinispan 8.2.0.

      Now and then our threads get stuck in state TIMED_WAITTING.

      Thread dump from this thread looks following:

       

      at sun.misc.Unsafe.park(Native Method)

      - parking to wait for <14cc5f> (a java.util.concurrent.CompletableFuture$Signaller)

      at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)

      at java.util.concurrent.CompletableFuture$Signaller.block(Unknown Source)

      at java.util.concurrent.ForkJoinPool.managedBlock(Unknown Source)

      at java.util.concurrent.CompletableFuture.timedGet(Unknown Source)

      at java.util.concurrent.CompletableFuture.get(Unknown Source)

      at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:100)

      at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:205)

      at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:162)

      at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:113)

      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

      at org.infinispan.interceptors.DeadlockDetectingInterceptor.handleDataCommand(DeadlockDetectingInterceptor.java:98)

      at org.infinispan.interceptors.DeadlockDetectingInterceptor.visitLockControlCommand(DeadlockDetectingInterceptor.java:80)

      at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:113)

      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

      at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

      at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)

      at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:113)

      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

      at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

      at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)

      at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:113)

      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

      at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.acquireRemoteIfNeeded(PessimisticLockingInterceptor.java:225)

      at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataWriteCommand(PessimisticLockingInterceptor.java:124)

      at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitPutKeyValueCommand(AbstractTxLockingInterceptor.java:63)

      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:78)

      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

      at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:367)

      at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:221)

      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:78)

      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

      at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

      at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:43)

      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:78)

      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

      at org.infinispan.statetransfer.StateTransferInterceptor.handleTxWriteCommand(StateTransferInterceptor.java:297)

      at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:279)

      at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:107)

      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:78)

      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

      at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:191)

      at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:177)

      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:78)

      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

      at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)

      at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:79)

      at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:43)

      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:78)

      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

      at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

      at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:43)

      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:78)

      at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:335)

      at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1672)

      at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:1121)

      at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1111)

      at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:453)


      It seems that it is stacked in Objects.requireNonNull(future, "Completable Future must be non-null.").get(BIG_DELAY_NANOS, TimeUnit.NANOSECONDS);


      My question here would be?

      - why it is stuck in there? what could cause this?

      - timeout BIG_DELAY_NANOS = TimeUnit.DAYS.toNanos(1) is huge. Is there reason why it is so big? What if it would be smaller (e.g. 5 min)?

      - what can we do to avoid such blocking state? Can some settings of more threads do the job?

       

      Can this be somehow connected to Show stopper: Infinispan hot rod server gets stuck / dead lock in high load with registered client listener in hot rod client - infinispan-server-8.2.1.Final? In fact we are not using hot rod client. Could issue be similar even when we are using embedded client?

       

      Any help is appreciated.

      Thanks

      Tomas