4 Replies Latest reply on Jul 29, 2015 4:00 AM by rvansa

Is Infinispan a good fit for this use case?

john.sanda Jul 28, 2015 10:00 AM

I work on Hawkular Metrics (H-Metrics) which is basically a time series database built on top of Cassandra. There are different situations in which we generate new time series from one or more existing time series. This will be done as one or more background jobs that run periodically. For example, H-Metrics defines a counter metric type. We automatically compute and store the rate for counters as new time series. This is done in a background job that runs on pre-configured interval, maybe every minute. In the worst case scenario, we have to perform N reads where N is the number of counter metrics. Cassandra is very scalable; so, we can add (Cassandra) nodes as necessary to deal with the read load, but this does not reduce the amount of I/O we have to do. We can do some schema optimizations that will reduce the overall number of reads. That might be an adequate approach, but I am curious about whether or not Infinispan would be a good fit in this situation.

Here is one approach I have considered. I maintain a separate cache per minute since I want to calculate/update rates every minute. When I receive a request to store counter data points, I write the data points both to Cassandra and to the cache for the current minute. The cache entry key would be the metric id, and the value would be the metric data point value. In this particular case I only need to store the latest value for the current minute, so I can simply replace existing cache entries. The background job that runs every minute would no longer query Cassandra for counter data points. It might instead kick off a DistributedExecutorService task that calculates the rates for each of the cache entries, writes them to Cassandra, and the finally deletes the cache. In this example, I simply inserted (or replaced) cache entries. For other metric data types, I will need to update existing cache entries.

The appeal of this approach is that I completely eliminate having to query Cassandra when calculating new time series data, but I do not have enough experience with Infinispan to know whether or not this would be a sound approach. Here are some additional questions I am considering.

Will the performance of lots of concurrent reads/writes on a cache be adequate such that it won't be a bottleneck when storing data points? Currently, the write path for storing data points only involves writing to Cassandra which is very fast. With a caching layer, it could also involve writing new cache entries and/or updating existing cache entries. For a local cache when all entries are in memory, I would expect cache operations to extremely fast as I am performing simple updates on the entries.

What about when the cache is distributed? For the moment, assume I am running multiple H-Metrics servers with embedded, distributed caches. If server-1 receives a request to store data and the cache for that data resides on server-2, do I need to be concerned latency of performing cache updates?

I previously mentioned using a separate cache for each minute. A new cache is created whenever I receive a metric data point having a timestamp for the next minute. Consider again a distributed cache with multiple H-Metrics servers. What if two H-Metrics servers both try to create the cache for the next minute at the same time. Do I need to be concerned about race conditions? Is cache creation fast? I am curious about how fast it is as the number of ISPN nodes increases.

When nodes are added/removed, what kind of performance impact is there data being reshuffled? Can I continue to serve client requests when a node is added/removed, or should this be done as an offline, maintenance operation?

So far I have only discussed using embedded caches? What are the pros/cons of using the HotRod server instead?

1. Re: Is Infinispan a good fit for this use case?

rvansa Jul 28, 2015 12:21 PM (in response to john.sanda)

I wouldn't create a cache for every minute, that would have a large overhead, IMO - in SQL database you wouldn't create new tables either. Instead, composite keys would do better work. When executing Map Reduce (I think that's more suitable for per-entry processing than DistExec) you can filter only particular entries.

As for the writes in distributed mode: Infinispan provides synchronous and asynchronous caches, therefore the write does not need to block the writer, if you're OK with reading outdated values for a while. The writes are automatically ordered, therefore older write will never overwrite newer one (with respect to the same writing node).

Nodes added/removed: If the nodes are added/removed programmatically, there shouldn't be any downtime - we have what we call non-blocking state transfer. Something seen as downtime could occur when you just kill the node (or pull the network plug), there the other nodes give wait a while until declaring that node dead (as GCs look the same way as node crash), and writes that should be handled by the killed node are blocked in the meantime.

Hot Rod should be used when application is often connecting/disconnecting, or you have just many apps and not too much data - in that case it is easier to have just few servers to maintain. However, Hot Rod will always lag in the feature list behind embedded mode.
Actions
2. Re: Is Infinispan a good fit for this use case?

john.sanda Jul 28, 2015 1:15 PM (in response to rvansa)

I wouldn't create a cache for every minute, that would have a large overhead, IMO - in SQL database you wouldn't create new tables either. Instead, composite keys would do better work. When executing Map Reduce (I think that's more suitable for per-entry processing than DistExec) you can filter only particular entries.

What kind of overhead is involved? With the approach I described, I would simply be able to put/replace cache entries without having to first perform any reads. With a composite key that consists of the metric id and a timestamp rounded down to the minute, I would have to perform read then write the cache entry.

I am not using an RDBMS. With Cassandra, I would't necessarily create new tables, but I might create new partitions for each minute. A partition is similar to a group of related rows in an RDBMS. All of the data points for the last minute (from multiple metrics) for instance, are stored across a handful of partitions. After I store the new data points, I can delete the partitions from the last minute. This is a fairly common data modeling approach in Cassandra. I might be trying to apply that way of thinking here. Maybe that is a mistake.

I think Map Reduce makes sense to me particularly with composite keys. Once I have generated the new data points and stored them in Cassandra, I want to delete the cache entries. What is the best to go about that?

As for the writes in distributed mode: Infinispan provides synchronous and asynchronous caches, therefore the write does not need to block the writer, if you're OK with reading outdated values for a while. The writes are automatically ordered, therefore older write will never overwrite newer one (with respect to the same writing node).

I do not need strong consistency here so the async writes would be a good fit.
Actions
3. Re: Is Infinispan a good fit for this use case?

rvansa Jul 29, 2015 4:00 AM (in response to john.sanda)
John Sanda wrote:

What kind of overhead is involved? With the approach I described, I would simply be able to put/replace cache entries without having to first perform any reads. With a composite key that consists of the metric id and a timestamp rounded down to the minute, I would have to perform read then write the cache entry.

The overhead is in starting a new cache, which involves creating a couple of new components on each node and exchanging messages to agree upon the topology, distribution of entries etc. Each such process could consume few threads blocked in some call (though this limitation should be removed in Infinispan 8.0), and when a new node is added, all this work needs to be performed for each cache. We have seen some difficulties when users tried to use thousands of caches. However, if you use each cache for just a short period of time, you could easily recycle them.

I don't really understand the read/write concern, IIUC you'd just do commonCache.put(new MetricAndTimestamp(metricId, timestamp), metricValue) in the same way as you'd call getCache("cache" + timestamp).put(metricId, metricValue);

I don't know Cassandra, so I can't tell you the exact match in terminology. As for any groups of rows, there's a grouping API to keep a set of keys on the same physical node, but that's not what you want here.

For multiRemove, you should either use MapReduce, or call
Cache asyncCache = cache.withFlags(Flag.FORCE_ASYNCHRONOUS, Flag.IGNORE_RETURN_VALUES); for (CacheEntry<K, ?> entry : cache.filterEntries(new MyFilter()).converter(new MyKeysOnlyConverter()) { asyncCache.remove(entry.getKey()); }

(If you already have the cache async, there's no need to use those flags) In Infinispan 8.0 there'll be a streaming API that could allow you to specify this through spliterators, too.
Actions
4. Re: Is Infinispan a good fit for this use case?

vdzhuvinov Jul 29, 2015 6:07 AM (in response to john.sanda)

John Sanda wrote:

I wouldn't create a cache for every minute, that would have a large overhead, IMO - in SQL database you wouldn't create new tables either. Instead, composite keys would do better work. When executing Map Reduce (I think that's more suitable for per-entry processing than DistExec) you can filter only particular entries.

What kind of overhead is involved? With the approach I described, I would simply be able to put/replace cache entries without having to first perform any reads. With a composite key that consists of the metric id and a timestamp rounded down to the minute, I would have to perform read then write the cache entry.

You could write a mini benchmark to measure what it takes takes to create a new Cache instance (which implements the java.util.Map interface). Creating a new cache is simply matter of calling the getCache method of org.infinispan.manager.EmbeddedCacheManager, provding the cache name as a parameter.

PS: Forgot to mention that the Cache instances will also need to be configured after they are created, e.g. for sync / async operation, etc. This can be done programmatically.
Actions

Go to original post