1 Reply Latest reply on Jun 23, 2016 2:58 PM by william.burns

    Threads blocked by Infinispan?

    tomas11
    tomas11 Jun 23, 2016 1:07 PM

      Hi all

       

      On our environment we have gone into following situation. This is taken out from thread dump.

       

      "OOB-264,testA,1005f36a-2860-4107-810a-d087224c9105-31733" - Thread t@786

         java.lang.Thread.State: RUNNABLE

          at org.nustaq.serialization.coders.FSTStreamEncoder.writeStringUTF(FSTStreamEncoder.java:305)

          at org.nustaq.serialization.FSTObjectOutput.writeObjectWithContext(FSTObjectOutput.java:392)

          at org.nustaq.serialization.FSTObjectOutput.writeObjectInternal(FSTObjectOutput.java:319)

          at org.nustaq.serialization.serializers.FSTMapSerializer.writeObject(FSTMapSerializer.java:49)

          at org.nustaq.serialization.FSTObjectOutput.writeObjectWithContext(FSTObjectOutput.java:456)

          at org.nustaq.serialization.FSTObjectOutput.writeObjectInternal(FSTObjectOutput.java:319)

          at org.nustaq.serialization.serializers.FSTMapSerializer.writeObject(FSTMapSerializer.java:49)

          at org.nustaq.serialization.FSTObjectOutput.writeObjectWithContext(FSTObjectOutput.java:456)

          at org.nustaq.serialization.FSTObjectOutput.writeObjectWithContext(FSTObjectOutput.java:361)

          at org.nustaq.serialization.FSTObjectOutput.writeObjectFields(FSTObjectOutput.java:639)

          at org.nustaq.serialization.FSTObjectOutput.defaultWriteObject(FSTObjectOutput.java:526)

          at org.nustaq.serialization.FSTObjectOutput.writeObjectWithContext(FSTObjectOutput.java:446)

          at org.nustaq.serialization.FSTObjectOutput.writeObjectInternal(FSTObjectOutput.java:319)

          at org.nustaq.serialization.FSTObjectOutput.writeObject(FSTObjectOutput.java:284)

          at org.nustaq.serialization.FSTObjectOutput.writeObject(FSTObjectOutput.java:193)

          at org.nustaq.serialization.FSTConfiguration.asByteArray(FSTConfiguration.java:1141)

          at com.serialization.AdvancedFastCustomSerializer.serialize(AdvancedFastCustomSerializer.java:74)

          at com.infinity.serialization.UniversalExternalizer.writeObject(UniversalExternalizer.java:64)

          at org.infinispan.marshall.core.ExternalizerTable$ForeignExternalizerAdapter.writeObject(ExternalizerTable.java:533)

          at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138)

          at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)

          at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)

          at org.infinispan.container.entries.ImmortalCacheValue$Externalizer.writeObject(ImmortalCacheValue.java:125)

          at org.infinispan.container.entries.ImmortalCacheValue$Externalizer.writeObject(ImmortalCacheValue.java:122)

          at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:484)

          at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138)

          at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)

          at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)

          at org.infinispan.remoting.responses.SuccessfulResponse$Externalizer.writeObject(SuccessfulResponse.java:84)

          at org.infinispan.remoting.responses.SuccessfulResponse$Externalizer.writeObject(SuccessfulResponse.java:77)

          at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:484)

          at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138)

          at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)

          at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)

          at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:72)

          at org.infinispan.marshall.core.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:77)

          at org.infinispan.commons.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:41)

          at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectToBuffer(AbstractDelegatingMarshaller.java:85)

          at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:23)

          at org.jgroups.blocks.RequestCorrelator.sendReply(RequestCorrelator.java:428)

          at org.jgroups.blocks.RequestCorrelator$ResponseImpl.send(RequestCorrelator.java:478)

          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.reply(CommandAwareRpcDispatcher.java:238)

          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.lambda$executeCommandFromLocalCluster$85(CommandAwareRpcDispatcher.java:195)

          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$$Lambda$20/10920806.reply(Unknown Source)

          at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:53)

          at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:132)

          at org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler.handle(NonTotalOrderTxPerCacheInboundInvocationHandler.java:99)

          at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)

          at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)

          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:196)

          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:166)

          at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:402)

          at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:352)

          at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:237)

          at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:695)

          at org.jgroups.JChannel.up(JChannel.java:738)

          at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030)

          at org.jgroups.protocols.RSVP.up(RSVP.java:201)

          at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)

          at org.jgroups.protocols.FlowControl.up(FlowControl.java:392)

          at org.jgroups.protocols.FlowControl.up(FlowControl.java:381)

          at org.jgroups.protocols.tom.TOA.up(TOA.java:121)

          at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1043)

          at org.jgroups.protocols.AUTH.up(AUTH.java:148)

          at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)

          at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1064)

          at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:779)

          at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)

          at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:652)

          at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)

          at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:200)

          at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)

          at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)

          at org.jgroups.protocols.Discovery.up(Discovery.java:296)

          at org.jgroups.protocols.TP.passMessageUp(TP.java:1590)

          at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1802)

          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

          at java.lang.Thread.run(Unknown Source)

       

         Locked ownable synchronizers:

          - locked <19ac84e> (a java.util.concurrent.ThreadPoolExecutor$Worker)

         

      "qtp15206738-32" - Thread t@32

         java.lang.Thread.State: TIMED_WAITING

          at sun.misc.Unsafe.park(Native Method)

          - parking to wait for <a89c58> (a java.util.concurrent.CompletableFuture$Signaller)

          at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)

          at java.util.concurrent.CompletableFuture$Signaller.block(Unknown Source)

          at java.util.concurrent.ForkJoinPool.managedBlock(Unknown Source)

          at java.util.concurrent.CompletableFuture.timedGet(Unknown Source)

          at java.util.concurrent.CompletableFuture.get(Unknown Source)

          at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:100)

          at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:205)

          at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.invokeClusterGetCommandRemotely(BaseDistributionInterceptor.java:188)

          at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.retrieveFromRemoteSource(BaseDistributionInterceptor.java:176)

          at org.infinispan.interceptors.distribution.TxDistributionInterceptor.remoteGet(TxDistributionInterceptor.java:344)

          at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitGetCommand(TxDistributionInterceptor.java:143)

          at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitGetKeyValueCommand(TxDistributionInterceptor.java:128)

          at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

          at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)

          at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.EntryWrappingInterceptor.visitDataReadCommand(EntryWrappingInterceptor.java:134)

          at org.infinispan.interceptors.EntryWrappingInterceptor.visitGetKeyValueCommand(EntryWrappingInterceptor.java:124)

          at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

          at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)

          at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:74)

          at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:78)

          at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:346)

          at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:331)

          at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

          at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)

          at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.statetransfer.StateTransferInterceptor.visitReadCommand(StateTransferInterceptor.java:177)

          at org.infinispan.statetransfer.StateTransferInterceptor.visitGetKeyValueCommand(StateTransferInterceptor.java:154)

          at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.CacheMgmtInterceptor.visitDataReadCommand(CacheMgmtInterceptor.java:103)

          at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:91)

          at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:79)

          at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)

          at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

          at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)

          at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)

          at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:335)

          at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:411)

          at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:403)

       

         Locked ownable synchronizers:

          - None


      Those are 2 threads which are hanging and not moving on. It looks like a deadlock?

      I am puzzeling over this and can not figure out what could be reason for that? What is worse so far only way how to get out of this situation is to restart a node - which is unpleasent.


      Do you see why this could happen? What more information should I provide to find out root cause.


      Any ideas are appreciated

      Thanks

      Tomas

        • 1. Re: Threads blocked by Infinispan?
          william.burns
          william.burns Jun 23, 2016 2:58 PM (in response to tomas11)

          Tomas,

           

          There is no deadlock shown in the 2 threads shown.

           

          I am guessing these threads are from 2 different nodes.  The first is a stack trace sending a response to a command (I am guessing a get command), but it is still serializing the results using a custom serialization framework.  The second is a user thread that is just blocking waiting for a response from a get command (this should time out with a replication exception though).

           

          Are you sure the first thread never progresses?  If not then it sounds like their may be a bug in the serialization of the object(s) you are sending.

           

          Are you sure the request doesn't eventually time out for the second thread?  If not maybe you have a very large replication timeout [1]?

           

          [1] ClusteringConfigurationBuilder (Infinispan JavaDoc 9.0.0.Alpha2 API)

            • Actions