Active invalidation of cache entries

[supinko]

Hello,

I currently have a local cache implementation based on BDBJE where my keys include a version number. I use that version number to manually delete cache entries (to free up disk space) when I know that they've been invalidated. The way I do this is to run a background thread that gets kicked off when the versions have potentially changed. This thread iterates through the cache's keyset and deletes outdated versions.

I've been trying to figure out how to translate this to Infinispan when I replace the local cache with a HotRod client. The documentation appears to strongly discourage the following pattern (which is basically my current implementation):

Iterator<MyKey> iter = cache.keySet().iterator();

while (iter.hasNext()) {

  MyKey key = iter.next();

  if (isOutdated(key)) {

    iter.remove();

  }

}

I would try something like

for (MyKey key : cache.keySet()) {

  if (isOutdated(key)) {

    VersionedValue val = cache.getVersioned();

    cache.removeWithVersion(key, val.getVersion());

  }

}

but (1) the documentation on keySet() says, "This method should only be used for debugging purposes such as to verify  that the cache contains all the keys   entered. Any other use involving execution of this method on a  production system is not recommended."  and (2) I don't really like the idea of fetching the data just to delete it.

Is there a better way to accomplish this? It would be great if there was a RemoteCache.getVersionedKeySet() that returned an immutable Set<VersionedKey<MyKey>>, so that I could do something like

for (VersionedKey<MyKey> vKey : cache.getVersionedKeySet()) {

  MyKey key = vKey.getKey();

  if (isOutdated(key)) {

    cache.removeWithVersion(key, vKey.getVersion());

  }

}

Thanks,

Supin

[supinko]

Or even better, maybe the Set<VersionedKey<MyKey>> that's returned by the mythical RemoteCache.getVersionedKeySet() could have an Iterator that allowed remove():

Iterator<VersionedKey<MyKey>> iter = cache.getVersionedKeySet();

while (iter.hasNext()) {

  VersionedKey<MyKey> vKey = iter.next();

  MyKey key = vKey.getKey();

  if (isOutdated(key)) {

    iter.remove();

  }

}

[manik]

Why do you need to iterate through the keys to delete them?  Can't you simply attach a lifespan to each entry so it is automatically removed after that lifespan has elapsed?  Or if you only want to remove the old version after a new version is added, then why not just use non-versioned keys, e.g.:

put(key, version1);

put(key, version2); // at this stage version1 will be overwritten

Cheers

Manik

[supinko]

Hi Manik,

The nature of our data is that we have relatively few, very large data sets (some being more than a GB) that are very expensive to calculate, but once calculated, they remain valid until someone changes the inputs for the original calculation. Since the data can be valid until the end of time, I can't use either a lifespan or an idle time expiration. Unfortunately, when someone makes a change, I don't have a dependency tree to tell me exactly what has been invalidated (it's likely that only a part of the cache is invalid, so I don't want to wipe the entire cache). Also, the data is lazily calculated, so when a change happens, we don't necessarily have a new version to replace an old version with right away; the cache usually ends up with lots of stale data that never gets overwritten.

I have enough information in my keys to determine whether an entry is valid or not, so that's why I took the approach of iterating through the keys.

Thanks,

Supin

[manik]

Then if your keys are small, maintain a separate list of keys in the cache?  You could then retrieve this key list to iterate over and remove entries you feel are invalid.