Index Consistency (for cache queries)

[paulpa63]

Hi,

As a result of reading a recent discussion: "Cache Store really supports transactional updates" I have been prompted to raise this discussion.  We are using indexes extensively in our application and have encountered a few issues (which have been fixed).  My outstanding fundamental question is: are Infinispan index updates transactional? i.e. is the Hibernate Search / Lucene indexing engine enlisted in the current transaction, when XA transactions have been enabled - so that there is a guarantee that cache updates and index content remain consistent?

Thanks in advance,

[sannegrinovero]

Short answer is no, but let me explain the full picture.

Indexes are updates in a post-transaction hook, which means that if the transaction on the cache is rolled back, no indexing operations happen. The limitation is that if indexing fails, then the  index state gets inconsistent with the data.

Currently this is not configurable and the design choice was made because:

• Index operations fail only in case of infrastructure failure: there are no locks, no conflicts possible, no validation triggers, etc..
• Indexing operations are expensive you want to postpone the operation as much as possible to keep the in-flight transactions short to minimize lock contention
• Lucene is not transactional: to make it behave as properly transactional would mean to completely serialize all operations
• Transactions guard the consistency of your data: an out-of-sync index can be rebuilt from the data, and this can be performed on the fly on a live system. Would you really want to fail a business transaction because of some index is having issues? Keeping in mind that "issues" likely means the disks storing the index are all dead, even reads are not going to happen so you might have some more serious problems.

Might be useful to know that the Search engine allows for registering of custom failure handlers: for example if consistency is more important than availability you could use it to take down the service in case the index is "destroyed" for some reason. I insist on terms like "destroyed" to stress that it must be something very critical going on to take it to fail an index operation, like multiple disks dead in an array (when storing on FSDirectory), or several nodes crashed (when using the Infinispan Directory, and all owners of distributed data crash at the same time).

I would like to explore improvements on this, but had limited time and it hasn't been a prority so far; if it's important in your case I would love to hear more about it. Suggestions welcome, but be warned all approaches propose so far would degrade performance very very significantly, and the practical likelyhood for this to be useful is not so clear.

There are two classes of solutions I have in mind so far:

1. develop a custom codec for Lucene which stores data in the cache in the same transaction -> rethink the whole index structure
2. design something which does not take advantage of Lucene indexes

Personally I think I'll be working on a custom codec (as there are other benefits to it as well), but to then have the index updates happen within the transaction scope of the data change will need more changes. Help or testers would be very welcome.

[paulpa63]

Hi Sanne,

Thanks for this information which includes a lot of interesting and illuminating detail.  Obviously I guessed the "short answer" but your justification of why things are as they are currently makes a lot of sense.

I have one thought / suggestion on this issue.  Would it be feasible to drop the concept of a seperate indexing sub-system completely and instead change the implementation of the query module to operate directly on the caches - making use of the distributed executor service where necessary?  This may require some modifications to internal structures and/or the use of second-level caches to optimise performance.  The design should not affect the performance of the cache updates very much - that processing should be simplified.  The challenge would likely be to maintain good performance from the query operations but these probably need to be accepted as heavier / slower operations anyway.  (I believe this strategy has been adopted by one of your competitor products).

Paul

[paulpa63]

Hi Sanne,

I think it is important to note one follow-up to the infornation / justifications you have provided on this issue, which relates directly back to my earlier discussion where I first asked this question about index consistency guarantees: at the moment in Infinispan 5.2.1, it is only possible to rebuild indexes on the fly (via the API) for distributed caches, see Question related to ISPN-1440 index rebuilding

So if you lose index consistency on a replicated cache you have got problems...

Paul