Local Transactional Cache looses data when eviction is enabled and there are multiple readers and one writer

[hchiorean]

Hi,

Using Infinispan 6.0.2 and a local, transactional cache backed by a <singleFile> store, with eviction enabled and a small "max-entries" setting, we have the following scenario:

• the main thread (i.e. the "writer") starts a transaction, adds a batch of strings into the cache and also appends the same strings into a List cache entry and then commits the transaction
• after the above has finished (i.e. after tx.commit) it fires a number of reader threads where each reader thread
  • a) checks that the string entries were added into the cache and
  • b) checks that the entries were correctly appended to the List entry
• the above steps are repeated a number of times

On any given run, based on timing, we're seeing that at some point (after some time) some of the reader threads will not see the latest version of the List entry (i.e. will not see the latest elements that were added into the list) but rather an old, stale List (sort of a "lost update" scenario).

If we either:

a) disable eviction or

b) set the max-entries to a large enough value (which I suspect has the same effect - not evicting anything) the problem doesn't show up.

What worries me from the ISPN docs is the following paragraph:

4.5. Cache Loaders and transactional caches

When a cache is transactional and a cache loader is present, the cache loader won’t be enlisted in the transaction in which the cache is part.

If this is correct, then I suspect that what's happening is that after tx.commit() finishes the eviction thread kicks in and removes the List entry from the cache, even though the persistent store has not yet been updated. When the next reader reads the List entry it will obviously get a stale entry from the store and bring it into memory. In effect this would mean that the [local cache,cache store] combination is not strongly consistent, which is a significant problem.

I'm attaching my ISPN configuration file and if required I can also provide a runnable piece of code that exhibits this problem.

Any suggestions are appreciated, tx

UPDATE: it turns out the cache store implementation is relevant as well: running with either H2 mem or file (via StringBasedJDBCCacheStore) I can't see the issue. However, using LevelDB the issue is there.

Message was edited by: Horia Chiorean

[rvansa]

The paragraph says that the cachestore wouldn't be *enlisted* in the transactions, therefore it won't participate in the 2PC protocol, changes cannot be rolled back etc. Instead, the write into the store will be executed during the second (commit) phase - after tx.commit() finishes, the entry should be persisted (although not fsynced).

With your TX settings, Infinispan is enlisted as synchronization (listener on the transaction), not a XA resource. It's possible that there's some error during the second (commit) phase and this won't be propagated to originator - please set <transaction  useSynchronization="false"/> (in Infinispan 7.0 this is FULL_XA or NON_DURABLE_XA).

[hchiorean]

Thanks for the update, I tested using <useSynchronization="false"/> and unfortunately it doesn't change the behavior.

I'm curious in general, since I'm dealing with a local cache running in a single process, why XA would come into play at all. AFAIK distributed transactions come into play only when there multiple network nodes.

[hchiorean]

Instead, the write into the store will be executed during the second (commit) phase - after tx.commit() finishes, the entry should be persisted (although not fsynced).

this got me thinking so I tested with other cachestores - StringBasedJDBC memory & LevelDB. It turns out I can't replicate the problem using StringBasedJDBC, but I can using LevelDB.

[rvansa]

XA doesn't have to do anything with distributed transactions - the transaction (in JTA sense) does not span across the cluster. I thought that something could be failing in the commit phase (where the store is written) and you wouldn't notice that, as exceptions in synchronization are not propagated.

You can't replicate with StringBasedJDBC cache store probably because it's orders of magnitude slower than single file store / leveldb.

Can you reproduce your issue with TRACE level logging on org.infinispan?

[hchiorean]

tried just now, but using TRACE on org.infinispan changes my timing sufficiently enough that I can't reproduce the issue. Is there a subpackage I can try this on ?

[hchiorean]

something else that's really important regarding the FS: I'm running this on a Windows 7 x64 NTFS FS.

[rvansa]

You can disable tracing for org.infinispan.marshall and org.infinispan.commons.marshall, but beyond that I can't give you hints what could be omitted without losing some vital info...

You can try to create a test case and then developers could take a look...