4 Replies Latest reply on Jun 22, 2016 8:50 AM by william.burns

    Concurrent TimeoutException when running multi clustered server

    haytham.salhi

      Hi,

       

      I am performing an experiment with the following details:

       

      1- I run 4 nodes as follows:

      ./standalone.sh -c clustered.xml -Djboss.node.name=nodeA -Djboss.socket.binding.port-offset=100

      ./standalone.sh -c clustered.xml -Djboss.node.name=nodeB -Djboss.socket.binding.port-offset=200

      ./standalone.sh -c clustered.xml -Djboss.node.name=nodeC -Djboss.socket.binding.port-offset=300

      ./standalone.sh -c clustered.xml -Djboss.node.name=nodeD -Djboss.socket.binding.port-offset=400

       

      And the cluster is setup without any issues.

       

      2- I initialize the default distributed cache with my own data (each entry represents an employee associated with organization) and everything works fine.

       

      3- In the client side, I run a somewhat large number of clients (about 64 clients). Each client open its own connection with the cluster. Then each client starts reading from that cache concurrently. At the beginning of the run, everything seems to work fine but after about (5-10) secs, the cluster above raises that something went wrong. For example, one node logs the exception below several times although I tried to increase the max-threads attribute as well as the replcation-timeout (don't know if they are related though) in /standalone/configuration/clustered.xml but it did not work. I attached the clustered.xml as well as the logs for all four nodes.

       

      Can anyone advise please? Why does the cluster crash when the number of clients is somewhat large? Is there a problem here or there?

       

      Exception

      http://127.0.0.1:10190/management

      [0m [0m16:05:58,877 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:10190

      [0m [0m16:05:58,878 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Infinispan Server 8.2.2.Final (WildFly Core 2.0.10.Final) started in 9186ms - Started 195 of 264 services (148 services are lazy, passive or on-demand)

      [0m [0m16:06:01,361 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,nodeB) ISPN000094: Received new cluster view for channel clustered: [nodeA|3] (4) [nodeA, nodeC, nodeB, nodeD]

      [0m [31m16:26:07,032 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (HotRodServerHandler-4-69) ISPN000136: Error executing command SizeCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException

        at org.infinispan.stream.impl.AbstractCacheStream.performOperationRehashAware(AbstractCacheStream.java:338)

        at org.infinispan.stream.impl.AbstractCacheStream.performOperation(AbstractCacheStream.java:258)

        at org.infinispan.stream.impl.AbstractCacheStream.performOperation(AbstractCacheStream.java:247)

        at org.infinispan.stream.impl.DistributedCacheStream.count(DistributedCacheStream.java:403)

        at org.infinispan.commands.read.SizeCommand.perform(SizeCommand.java:54)

        at org.infinispan.commands.read.SizeCommand.perform(SizeCommand.java:25)

        at org.infinispan.interceptors.CallInterceptor.handleDefault(CallInterceptor.java:125)

        at org.infinispan.commands.AbstractVisitor.visitSizeCommand(AbstractVisitor.java:80)

        at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

        at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

        at org.infinispan.commands.AbstractVisitor.visitSizeCommand(AbstractVisitor.java:80)

        at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

        at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

        at org.infinispan.commands.AbstractVisitor.visitSizeCommand(AbstractVisitor.java:80)

        at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

        at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

        at org.infinispan.commands.AbstractVisitor.visitSizeCommand(AbstractVisitor.java:80)

        at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

        at org.infinispan.statetransfer.StateTransferInterceptor.handleDefault(StateTransferInterceptor.java:379)

        at org.infinispan.commands.AbstractVisitor.visitSizeCommand(AbstractVisitor.java:80)

        at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

        at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

        at org.infinispan.commands.AbstractVisitor.visitSizeCommand(AbstractVisitor.java:80)

        at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

        at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:114)

        at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:83)

        at org.infinispan.commands.AbstractVisitor.visitSizeCommand(AbstractVisitor.java:80)

        at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

        at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

        at org.infinispan.commands.AbstractVisitor.visitSizeCommand(AbstractVisitor.java:80)

        at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)

        at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:335)

        at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:370)

        at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:365)

        at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:266)

        at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:112)

        at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$1(ContextHandler.java:56)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)

        at java.lang.Thread.run(Thread.java:745)

       

       

      [0m [31m16:26:07,031 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (HotRo