How to get cluster-wide statisctics in Infinispan

[ursudheesh]

Hi,

I am using Infinispan_5.0_Final and JDK1.6. I have written a standalone client for a distributed cache configuration to capture the cache

statistics using JMX.

I have enabled Cache level JMX configurations. After inserting an object into the cache and reading it, i could get the cache Hits counter as 1.

My code snippet is as follows:

                              MBeanServerConnection mBeanServer = ManagementFactory

                                                  .getPlatformMBeanServer();

                              String jmxQuery = "org.infinispan:type=Cache,name="+"\""+"InfinispanTest(dist_sync)"+"\""+

                                                                        ",manager="+"\""+"DefaultCacheManager"+"\""+

                                                      ",component=Statistics";

                              Set<ObjectInstance> queryResults;

                              try {

                                        queryResults = mBeanServer.queryMBeans( new ObjectName(jmxQuery), null);

                                        long cacheHits = 0;

                                        for (ObjectInstance objectInstance : queryResults) {

                                                  ObjectName objectName = objectInstance.getObjectName();

                                                   cacheHits = (Long) mBeanServer.getAttribute(objectName,  "Hits");

                              }catch(Exception e){

                              }

          }

My question is how to obtain the cluster wide cache statistics when multiple cahce instances are running.

Could you pls provide me with some example of code or configuration.

Regards

Sudheesh

[galder.zamarreno]

You'd need to loop through all MBeanServers that form the backend cluster.

[ursudheesh]

Hi Galder,

Thanks for your reply.

In our test scenario,  the nodes are within the local loop back i.e running on 127.0.0.1[7800], 127.0.0.1[7801].

The clients to both the nodes are doing 2 put and 1 get operations each into the cache.

This means in our scenario there is only one MBeanServer.

When i execute the first client, the "hits" count is 1. However when i execute the second client, I expect the "hits" count to be 2. But i am getting the hits as 1 only.

We have a StatisticsAgent class which implements Callable<Long> ,Externalizer<Long>. The call() override method will look up the mbeanserver for the the "hits" attribute and return its value.

At the client, will using DefaultExecutorService.invokeAll(Collections <StatisticsAgent>) get us an aggregated value of the "hits" attribute?

[galder.zamarreno]

Right, so if you have a single MBeanServer and you have two nodes running, you have to compile the stats from the mbeans belonging to the two caches. Information on the JMX naming conventions can be found in https://docs.jboss.org/author/display/ISPN/Management+Tooling#ManagementTooling-Infinispan4.2orlater

[ursudheesh]

I decided to the use DefaultExecutorService  to post the task of calculating the statistics on each of the Infinispan nodes.

My cache client is using the following snippet:

DefaultExecutorService service =  new DefaultExecutorService(cache);

StatisticsAgent agent = new StatisticsAgent(cache.getName(), cache.getConfiguration().getCacheMode().toString());

And the StatisticsAgent class is like :

public class StatisticsAgent  implements Callable<Long> ,Externalizer<Long>{

@Override

          public Long call() throws Exception {

long totalHits =0;

MBeanServerConnection mBeanServer = ManagementFactory

                    .getPlatformMBeanServer();

                    String jmxQuery = "org.infinispan:type=Cache,name="+"\""+cacheName+"("+cacheMode+")"+"\""+

                                          ",manager="+"\""+"DefaultCacheManager"+"\""+

                        ",component=Statistics";

                    Set<ObjectInstance> queryResults;

                      try {

                              queryResults = mBeanServer.queryMBeans(new ObjectName(jmxQuery), null);

                              for (ObjectInstance objectInstance : queryResults) {

                                        ObjectName objectName = objectInstance.getObjectName();

   long cacheHits = (Long) mBeanServer.getAttribute(objectName, "Hits");

   long removeHits =(Long) mBeanServer.getAttribute(objectName, "RemoveHits");

   long totalHits = cacheHits+removeHits;

                              }

                    } catch (Exception e) {

                              throw new RuntimeException(e.getMessage(), e);

                    }

                    return totalHits;

          }

}

}

[galder.zamarreno]

@Sudheesh, good stuff. I'm wondering what the cacheHits and removeHits addition is trying to represent exactly. Those two only cover get() and remove() calls.

[ursudheesh]

Galder,

          Are only cache get() operations considered for CacheHits count? Pls clarify. We added cacheHits & removeHits to reperesent overall cache operations. Is this correct?

          Another question related to computation of  statistics is while using replicated cache configuration and executing the above with 2 clients, i get the following error in the client invoking the DefaultExecutorService:

          java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Invalid response {}

            at org.infinispan.distexec.DefaultExecutorService$DistributedRunnableFuture.retrieveResult(DefaultExecutorService.java:582)

            at org.infinispan.distexec.DefaultExecutorService$DistributedRunnableFuture.get(DefaultExecutorService.java:529)

            at com.hp.ispan.jmx.test.MyTest.main(MyTest.java:53)

          Caused by: java.lang.IllegalStateException: Invalid response {}

            ... 3 more

Can we use DefaultExecutorService or  DistributedRunnableFuture in a replicated cache scenario? Pls confirm

Is there any other work around to obtain statistics for replicated cache.

[galder.zamarreno]

Yup, cache hits only count for get operations that returned something other than null, see https://github.com/galderz/infinispan/blob/master/core/src/main/java/org/infinispan/interceptors/CacheMgmtInterceptor.java for more info. You might also want to take into account the misses and the store calls. Depends on your use case...

That exception is missing where the ''Invalid response {}" is coming from, so can you please attach the entire stacktrace?

I don't see why it should not work for replicated caches. If it's not working, it could be a bug.

[ursudheesh]

Galder,

I have configured the cache for replication mode.

<clustering mode="replication"> <stateRetrieval timeout="20000" fetchInMemoryState="true"

                              alwaysProvideInMemoryState="true" /> <sync replTimeout="30000" /> </clustering>

I have 2 clients accessing the same cache.

1. The first client (MyTestAgain.java) has 2 put and 1 get operation

2. The second client (MyTest.java) has 2 put and 2 get operation and also the call to the statisticsAgent implementation.(as mentioned previsouly)

First Client is :

public class MyTestAgain {

          public static void main(String[] args) {

                              EmbeddedCacheManager manager;

                              try {

                                        manager = new DefaultCacheManager("InfinispanCfgForNode1.xml");

                                        Cache cache = manager.getCache("InfinispanClientTest");

                                        cache.put("LC", "Cat");

                                        cache.put("PB", "Pat");

                                        cache.get("PB");

                              } catch (IOException e) {

                                        e.printStackTrace();

                              }

          }

}

Second Client is :

public class MyTest {

          public static void main(String[] args) {

                    // TODO Auto-generated method stub

                    try {

                              EmbeddedCacheManager manager = new DefaultCacheManager("InfinispanCfgForNode1.xml");

                              Cache cache = manager.getCache("InfinispanClientTest");

                              cache.put("V", "Vidya");

                              cache.put("P", "Sudheesh");

                              cache.get("P");

                              cache.get("V");

                              DefaultExecutorService service =  new DefaultExecutorService(cache);

                              StatisticsAgent agent = new StatisticsAgent(cache.getName(), cache.getConfiguration().getCacheModeString());

                              try {

                                                  long cacheHits =0;

                                                  List<Future<Long>> futures = service.submitEverywhere(agent);

                                                  for(Future<Long> future : futures){

                                                            System.out.println("*************"+future);

                                                            System.out.println("future.get = " + future.get());

                                                      cacheHits += (Long)future.get();

                                                      System.out.println("Total Hits is  = " + cacheHits);

                                                  }

                              } catch (InterruptedException e) {

                                        // TODO Auto-generated catch block

                                        e.printStackTrace();

                              } catch (ExecutionException e) {

                                        // TODO Auto-generated catch block

                                        e.printStackTrace();

                              }

                              service.shutdown();

                    } catch (IOException e) {

                              // TODO Auto-generated catch block

                              e.printStackTrace();

                    }

          }

}

I get the exception while doing  at line future.get()

This error is occuring when i run the second client while the first one is still running. This error will not occur when the second client is run as stand alone