Local Transactional Cache looses data when eviction is enabled and there are multiple readers and one writerhchiorean Oct 1, 2014 7:34 AM
Using Infinispan 6.0.2 and a local, transactional cache backed by a <singleFile> store, with eviction enabled and a small "max-entries" setting, we have the following scenario:
- the main thread (i.e. the "writer") starts a transaction, adds a batch of strings into the cache and also appends the same strings into a List cache entry and then commits the transaction
- after the above has finished (i.e. after tx.commit) it fires a number of reader threads where each reader thread
- a) checks that the string entries were added into the cache and
- b) checks that the entries were correctly appended to the List entry
- the above steps are repeated a number of times
On any given run, based on timing, we're seeing that at some point (after some time) some of the reader threads will not see the latest version of the List entry (i.e. will not see the latest elements that were added into the list) but rather an old, stale List (sort of a "lost update" scenario).
If we either:
a) disable eviction or
b) set the max-entries to a large enough value (which I suspect has the same effect - not evicting anything) the problem doesn't show up.
What worries me from the ISPN docs is the following paragraph:
4.5. Cache Loaders and transactional caches
When a cache is transactional and a cache loader is present, the cache loader won’t be enlisted in the transaction in which the cache is part.
If this is correct, then I suspect that what's happening is that after tx.commit() finishes the eviction thread kicks in and removes the List entry from the cache, even though the persistent store has not yet been updated. When the next reader reads the List entry it will obviously get a stale entry from the store and bring it into memory. In effect this would mean that the [local cache,cache store] combination is not strongly consistent, which is a significant problem.
I'm attaching my ISPN configuration file and if required I can also provide a runnable piece of code that exhibits this problem.
Any suggestions are appreciated, tx
UPDATE: it turns out the cache store implementation is relevant as well: running with either H2 mem or file (via StringBasedJDBCCacheStore) I can't see the issue. However, using LevelDB the issue is there.
Message was edited by: Horia Chiorean