We are using Infinispan as a distributed cache. When we enable Infinispan in one node only and send requests through 40 client threads. Our single operation is combination of get+put requests.
The setup consists of two cache 1st is Dist Sync Cache with number of owners as 2 .
Second is in replicated mode.
The final intention is to have 3 servers with these two cache.
It gives throughput of 39-40 k (get + put) in Dist Sync Cache with number of owners as 2 and also internally it does 40k puts for REPL_SYNC cache which is replicated mode. So total 120k transactions.per sec across all threads.
When same setup is replicated in two nodes in same machine. The overall speed becomes 21k when complared to 120k . Which is 1/5th of original.
When same setup is replicated in two nodes in different machines.The overall speed becomes 7k when complared to 120k . Which is 1/20th of original.
We are using TCP for communication between nodes. The jgroups_tcp.xml is attached. The put and get operations are getWithVersion and replacewithVersion for main cache from client end.
For secondary cache its simple put and get operation. Doing Flag.SKIP_CACHE_LOAD along with put operation in second cache improved speed a bit but over all we are expecting the distributed cache to behave with around 90k per sec kind of speed if not mode when we are sharing it across multiple nodes . But presently the speeds are too low for production deployment.
Any ideas would be quite helpful.