0 Replies Latest reply on May 9, 2013 11:23 AM by Balazs Zsoldos

    Why is there much more network traffic when I create more than two clustered cache than with two

    Balazs Zsoldos Newbie

      I created a test app based on https://docs.jboss.org/author/display/ISPN/Using+Infinispan+as+a+JCache+provider#UsingInfinispanasaJCacheprovider-ClusteringJCacheinstances.

       

      I also created a stress test that:

      • Inserts data
        • On 100 threads 1000 times on each thread
        • Check if data is available on cache2
      • Read all the inserted data in one thread from cache2
      • Delete all the data from cache2

       

      If I use two cache istances there is no network traffic (1-2 KB/s). If I use three or more cache instances there is arouind 500-1500 KB/s upload network traffic. Could you please tell me where the huge difference comes from?

       

      When I tried the same with distribution caches it did not even start with more than two caches. About that I created a JIRA issue.

       

      Some performance comparison

       

      2 cache instances


       

      Threads * Iteration (2 nodes)

      Insert time

      Query time

      Delete time

      100 threads * 1.000 iteration

      ~ 12,5 s

        ~ 90 ms

      ~ 3,1 s

      10 threads * 10.000 iteration

      ~ 9,4 s

      ~ 90 ms

      ~ 3,5 s

      100 threads * 10.000 iteration

      ~ 40 s

      ~ 650 ms

      ~25 s

       

      3 Cache instances


       

      Threads * Iteration (3 nodes)

      Insert time

      Query time

      Delete time

      100 threads * 1.000 iteration

      ~ 40 s

        ~ 90 ms

      ~ 24 s

      10 threads * 10.000 iteration

      ~ 38 s

      ~ 90 ms

      ~ 8,5 s

      100 threads * 10.000 iteration

      ~ -- s

      ~ -- ms

      ~ -- s

       

      I did not write time for "100 threads * 10000 iteration" As after a while it stopped working with the following stacktrace:

       

      Exception in thread "Thread-13" org.infinispan.util.concurrent.TimeoutException: Replication timeout for localhost-11731

          at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:110)

          at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:554)

          at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:303)

          at org.infinispan.interceptors.ReplicationInterceptor.handleCrudMethod(ReplicationInterceptor.java:319)

          at org.infinispan.interceptors.ReplicationInterceptor.visitPutKeyValueCommand(ReplicationInterceptor.java:270)

          at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:82)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

          at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:302)

          at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:157)

          at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:82)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

          at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitPutKeyValueCommand(NonTransactionalLockingInterceptor.java:73)

          at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:82)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)

          at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)

          at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:82)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)

          at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)

          at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:82)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

          at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)

          at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194)

          at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136)

          at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:82)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)

          at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)

          at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:82)

          at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)

          at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1251)

          at org.infinispan.CacheImpl.putInternal(CacheImpl.java:801)

          at org.infinispan.CacheImpl.put(CacheImpl.java:787)

          at org.infinispan.DecoratedCache.put(DecoratedCache.java:327)

          at org.infinispan.jcache.JCache.put(JCache.java:692)

          at org.infinispan.jcache.JCache.doPut(JCache.java:482)

          at org.infinispan.jcache.JCache.put(JCache.java:447)

          at hu.nticket.test.infinispan.Main$1.run(Main.java:59)

          at java.lang.Thread.run(Thread.java:722)

       

      Could you please let me know if there is a big code difference if there are 2 or more nodes?

      I ran the tests many times but I had a feeling that the execution time is not really constant.

       

      Could you please let me know if this problem only experienced on my side or you are working on this to be more stable?

       

      In case you are interested please find attached the stress tests I worked with.