Why is there much more network traffic when I create more than two clustered cache than with two

[zs_b]

I created a test app based on https://docs.jboss.org/author/display/ISPN/Using+Infinispan+as+a+JCache+provider#UsingInfinispanasaJCacheprovider-ClusteringJCacheinstances.

I also created a stress test that:

• Inserts data
  • On 100 threads 1000 times on each thread
  • Check if data is available on cache2
• Read all the inserted data in one thread from cache2
• Delete all the data from cache2

If I use two cache istances there is no network traffic (1-2 KB/s). If I use three or more cache instances there is arouind 500-1500 KB/s upload network traffic. Could you please tell me where the huge difference comes from?

When I tried the same with distribution caches it did not even start with more than two caches. About that I created a JIRA issue.

Some performance comparison

2 cache instances

Threads * Iteration (2 nodes)	Insert time	Query time	Delete time
100 threads * 1.000 iteration	~ 12,5 s	~ 90 ms	~ 3,1 s
10 threads * 10.000 iteration	~ 9,4 s	~ 90 ms	~ 3,5 s
100 threads * 10.000 iteration	~ 40 s	~ 650 ms	~25 s

3 Cache instances

Threads * Iteration (3 nodes)	Insert time	Query time	Delete time
100 threads * 1.000 iteration	~ 40 s	~ 90 ms	~ 24 s
10 threads * 10.000 iteration	~ 38 s	~ 90 ms	~ 8,5 s
100 threads * 10.000 iteration	~ -- s	~ -- ms	~ -- s

I did not write time for "100 threads * 10000 iteration" As after a while it stopped working with the following stacktrace:

Exception in thread "Thread-13" org.infinispan.util.concurrent.TimeoutException: Replication timeout for localhost-11731

    at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:110)

    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:554)

    at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:303)

    at org.infinispan.interceptors.ReplicationInterceptor.handleCrudMethod(ReplicationInterceptor.java:319)

    at org.infinispan.interceptors.ReplicationInterceptor.visitPutKeyValueCommand(ReplicationInterceptor.java:270)

    at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:82)

    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

    at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:302)

    at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:157)

    at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:82)

    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

    at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitPutKeyValueCommand(NonTransactionalLockingInterceptor.java:73)

    at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:82)

    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

    at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)

    at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)

    at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:82)

    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

    at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)

    at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)

    at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:82)

    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

    at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)

    at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194)

    at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136)

    at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:82)

    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

    at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)

    at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)

    at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)

    at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:82)

    at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)

    at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1251)

    at org.infinispan.CacheImpl.putInternal(CacheImpl.java:801)

    at org.infinispan.CacheImpl.put(CacheImpl.java:787)

    at org.infinispan.DecoratedCache.put(DecoratedCache.java:327)

    at org.infinispan.jcache.JCache.put(JCache.java:692)

    at org.infinispan.jcache.JCache.doPut(JCache.java:482)

    at org.infinispan.jcache.JCache.put(JCache.java:447)

    at hu.nticket.test.infinispan.Main$1.run(Main.java:59)

    at java.lang.Thread.run(Thread.java:722)

Could you please let me know if there is a big code difference if there are 2 or more nodes?

I ran the tests many times but I had a feeling that the execution time is not really constant.

Could you please let me know if this problem only experienced on my side or you are working on this to be more stable?

In case you are interested please find attached the stress tests I worked with.