Performance/tuning of Infinispan as a distributed cache

[mike-evans]

We're evaluating Infinispan as a possible replacement for memcached as a store for frequently updated location data in Project Clearwater (www.projectclearwater.org).  We run the memcached cluster on the same systems as some of our message processing function, and currently memcached takes about 15% of CPU on an EC2 m1.small at peak load.  The rationale for switching to Infinispan is for the replication function and the ability to grow and shrink the cluster without losing information.  However, our initial performance tests suggest Infinispan will take significantly more CPU than memcached (around 75%), almost certainly more than we can afford to give it, so I wanted to see if there is anything we can do to tune the Infinispan configuration we are running to improve the situation.

The test configuration is a simple two node distributed cache, using JGroups TCP transport, with mode="ASYNC" and owners="2" in the cache configuration (I'll attach the full config file).  The test load is a read/write-with-cas cycle using the memcached ASCII protocol, with 50 threads running in parallel in the test application.  All read/writes are sent to one node in the cluster, and we're measuring CPU utilisation on both nodes (the test application is running on a separate system).  The results we see are

• ~1500 r/w operations per second in total
• CPU utilisation is ~85% on the node handling the read/write commands and ~45% on the other node

For info, we've also run a read-only workload, where we see throughput of ~5600 reads/second with CPU utilisation of 75% on the node handling the reads and 0% on the other node.

So, as I said above, are there any configuration changes we can make to optimize this kind of work load?

[mike-evans]

I forgot to add, we are running Ubuntu 12.04, and using OpenJDK 6.

I should also explain how the CPU numbers above are calculated.  The numbers are extrapolated from mpstat output, by summing everything apart from stea%l and guest% then scaling this total based on the peak CPU utilisation we typically see on an EC2 m1.small.  We usually find that when our applications are using around 45% the idle CPU has dropped close to zero and steal is taking the majority of the remaining CPU presumably because the hypervisor is limiting the CPU available to our instance.

[mircea.markus]

I'd expect Infinispan to take more CPU but the difference is way too high.

Mind creating a JIRA for this? https://issues.jboss.org/browse/ISPN

Also if you have some profiler snapshot at hand with thread activity would be awesome :-)

[mike-evans]

I thought it would be premature to raise a JIRA until I was sure that I have the configuration right and that my expectation that the performance should be close to memcached was reasonable.  Is there anything I should change in the configuration that would potentially improve things?

I don't have profiler output at the moment, but I'll look at setting this up.

[mircea.markus]

Is there anything I should change in the configuration that would potentially improve things?

the config looks good.