Threads blocked by Infinispan?
tomas11 Jun 23, 2016 1:07 PMHi all
On our environment we have gone into following situation. This is taken out from thread dump.
"OOB-264,testA,1005f36a-2860-4107-810a-d087224c9105-31733" - Thread t@786
java.lang.Thread.State: RUNNABLE
at org.nustaq.serialization.coders.FSTStreamEncoder.writeStringUTF(FSTStreamEncoder.java:305)
at org.nustaq.serialization.FSTObjectOutput.writeObjectWithContext(FSTObjectOutput.java:392)
at org.nustaq.serialization.FSTObjectOutput.writeObjectInternal(FSTObjectOutput.java:319)
at org.nustaq.serialization.serializers.FSTMapSerializer.writeObject(FSTMapSerializer.java:49)
at org.nustaq.serialization.FSTObjectOutput.writeObjectWithContext(FSTObjectOutput.java:456)
at org.nustaq.serialization.FSTObjectOutput.writeObjectInternal(FSTObjectOutput.java:319)
at org.nustaq.serialization.serializers.FSTMapSerializer.writeObject(FSTMapSerializer.java:49)
at org.nustaq.serialization.FSTObjectOutput.writeObjectWithContext(FSTObjectOutput.java:456)
at org.nustaq.serialization.FSTObjectOutput.writeObjectWithContext(FSTObjectOutput.java:361)
at org.nustaq.serialization.FSTObjectOutput.writeObjectFields(FSTObjectOutput.java:639)
at org.nustaq.serialization.FSTObjectOutput.defaultWriteObject(FSTObjectOutput.java:526)
at org.nustaq.serialization.FSTObjectOutput.writeObjectWithContext(FSTObjectOutput.java:446)
at org.nustaq.serialization.FSTObjectOutput.writeObjectInternal(FSTObjectOutput.java:319)
at org.nustaq.serialization.FSTObjectOutput.writeObject(FSTObjectOutput.java:284)
at org.nustaq.serialization.FSTObjectOutput.writeObject(FSTObjectOutput.java:193)
at org.nustaq.serialization.FSTConfiguration.asByteArray(FSTConfiguration.java:1141)
at com.serialization.AdvancedFastCustomSerializer.serialize(AdvancedFastCustomSerializer.java:74)
at com.infinity.serialization.UniversalExternalizer.writeObject(UniversalExternalizer.java:64)
at org.infinispan.marshall.core.ExternalizerTable$ForeignExternalizerAdapter.writeObject(ExternalizerTable.java:533)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138)
at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)
at org.infinispan.container.entries.ImmortalCacheValue$Externalizer.writeObject(ImmortalCacheValue.java:125)
at org.infinispan.container.entries.ImmortalCacheValue$Externalizer.writeObject(ImmortalCacheValue.java:122)
at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:484)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138)
at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)
at org.infinispan.remoting.responses.SuccessfulResponse$Externalizer.writeObject(SuccessfulResponse.java:84)
at org.infinispan.remoting.responses.SuccessfulResponse$Externalizer.writeObject(SuccessfulResponse.java:77)
at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:484)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138)
at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)
at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:72)
at org.infinispan.marshall.core.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:77)
at org.infinispan.commons.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:41)
at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectToBuffer(AbstractDelegatingMarshaller.java:85)
at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:23)
at org.jgroups.blocks.RequestCorrelator.sendReply(RequestCorrelator.java:428)
at org.jgroups.blocks.RequestCorrelator$ResponseImpl.send(RequestCorrelator.java:478)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.reply(CommandAwareRpcDispatcher.java:238)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.lambda$executeCommandFromLocalCluster$85(CommandAwareRpcDispatcher.java:195)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$$Lambda$20/10920806.reply(Unknown Source)
at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:53)
at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:132)
at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)
at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:196)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:166)
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:402)
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:352)
at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:237)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:695)
at org.jgroups.JChannel.up(JChannel.java:738)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030)
at org.jgroups.protocols.RSVP.up(RSVP.java:201)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:392)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:381)
at org.jgroups.protocols.tom.TOA.up(TOA.java:121)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1043)
at org.jgroups.protocols.AUTH.up(AUTH.java:148)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1064)
at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:779)
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:652)
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:200)
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
at org.jgroups.protocols.Discovery.up(Discovery.java:296)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1590)
at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1802)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked <19ac84e> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"qtp15206738-32" - Thread t@32
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <a89c58> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.CompletableFuture$Signaller.block(Unknown Source)
at java.util.concurrent.ForkJoinPool.managedBlock(Unknown Source)
at java.util.concurrent.CompletableFuture.timedGet(Unknown Source)
at java.util.concurrent.CompletableFuture.get(Unknown Source)
at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:100)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:205)
at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.invokeClusterGetCommandRemotely(BaseDistributionInterceptor.java:188)
at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.retrieveFromRemoteSource(BaseDistributionInterceptor.java:176)
at org.infinispan.interceptors.distribution.TxDistributionInterceptor.remoteGet(TxDistributionInterceptor.java:344)
at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitGetCommand(TxDistributionInterceptor.java:143)
at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitGetKeyValueCommand(TxDistributionInterceptor.java:128)
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.EntryWrappingInterceptor.visitDataReadCommand(EntryWrappingInterceptor.java:134)
at org.infinispan.interceptors.EntryWrappingInterceptor.visitGetKeyValueCommand(EntryWrappingInterceptor.java:124)
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:74)
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:78)
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:346)
at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:331)
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.statetransfer.StateTransferInterceptor.visitReadCommand(StateTransferInterceptor.java:177)
at org.infinispan.statetransfer.StateTransferInterceptor.visitGetKeyValueCommand(StateTransferInterceptor.java:154)
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.CacheMgmtInterceptor.visitDataReadCommand(CacheMgmtInterceptor.java:103)
at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:91)
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:79)
at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:335)
at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:411)
at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:403)
Locked ownable synchronizers:
- None
Those are 2 threads which are hanging and not moving on. It looks like a deadlock?
I am puzzeling over this and can not figure out what could be reason for that? What is worse so far only way how to get out of this situation is to restart a node - which is unpleasent.
Do you see why this could happen? What more information should I provide to find out root cause.
Any ideas are appreciated
Thanks
Tomas