Hibernate/Cache conceptual mismatch

[sebersole]

One of the things we see a lot in the context of usage from Hibernate is contention due to puts into the cache which logically are not writes. The scenario I am talking about is when Hibernate loads some data from the database and then places that into the second level cache. After Hibernate loads this data from the database, it then does a put into the cache. JBossCache interprets that as a write and issues a write lock.

This is what I see as the referenced mismatch. Cache considers itself as the "truth" of the system so far as data state is concerned. However, at least in this usage, that is clearly not the case; the database is the "truth".

What really *should* happen in this scenario is for JBossCache to instead acquire a read lock on the created node (unless the user specifies some form of write lock in the db, in which case we may want to also acquire a write lock in the cache; but that's a whole other discussion).

So I have given some thought about how to best resolve this. I first thought through using current JBossCache concepts. Like loaders. I think loaders might actually work, except for the fact that they are static for the whole cache instance; for that approach to work for us, we would really need to be able to specify a loader *per invocation*.

Next (and to my mind the most viable option) was to introduce a "load' semantic to the API. This is similar to a put, except that here we force JBossCache to only acquire a read lock instead of the write lock.

Thoughts?

[brian.stansberry]

Is there any time when a write lock on the cache is necessary?

To express the question more clearly, in the Hibernate use case is there a need for locking semantics on the cache stronger than READ_UNCOMMITTED? If not, perhaps READ_UNCOMMITTED would work.

It wouldn't work if the contention you're talking about is between writes.

[sebersole]

What I am saying is that there is two usage patterns which force Hibernate to do puts into the cache. But to JBossCache, a put is a put is a put and as such always receives a write lock; it does not distinguish the purpose or the context of the put.

(1) Hibernate loads data from the database and perfoms a put into the cache. This is the one discussed before. This is logically just a read. We are not putting a change of system state into the cache. Thus this should be a read lock to minimize contention.
(2) User, via Hibernate, modifies some data requiring Hibernate to do a put into the cache to store the new state. This should be write locked.

[brian.stansberry]

OK, that's clear. Figured it couldn't be that easy, but had to ask :)

Not sure what you mean about the loader idea as locking and cache loading are separate aspects, but in any case the load semantic in the API sounds cleaner. Manik can better comment on this, but there's a big drive in 2.0 to reduce the number of methods in the API, so an alternative is some kind of Option to only use the read lock.

[ben.wang]

I think what you describe is a mismatch indeed. From Cache point of view, when you deposit any data, if there is concurrency, then some kind of lokc is necessary. So I think what you are looking for is lock level that can be changed per API call. Something like Brian suggested to put into Option.

However, it can be tricky though. Just imagine that I have /a/b, and /a/c. If node "b" and "c" have different lock levels during a tx, then there can be data integrity issue.

Steve, is this mismatch causing deadlock or performance slowdown?

[sebersole]

Highly likely.

Consider 2 different Hibernate Sessions performing the same load concurrently:

session.load( MyEntity.class, new Long(1) );

And assume that MyEntity#1 is not currently in the cache. So what happens?

Well, in response to the load from the first session Hibernate will load the data from the database and then perform the put into the cache, acquiring: (1) a read lock in the database and (2) a write lock in the cache.

Now, the second load request gets processed. So Hibernate checks the cache to see if the data exists there. However, we have previously acquired a write lock in the other session so this process is blocked until that write lock is released when the first session ends its transaction.

[ben.wang]

For the case of querying/loading, if Cache does not participate in the tx, then I think it wont probably get deadlock and the response time will be much faster as well? Am I correct?

[manik]

Now, the second load request gets processed. So Hibernate checks the cache to see if the data exists there. However, we have previously acquired a write lock in the other session so this process is blocked until that write lock is released when the first session ends its transaction.

I thought you use the fail silently Option when interacting with the cache? This will prevent the cache from participating in any ongoing tx so the write lock will only exist for the duration of the cache.put() method call, not the entire transaction.

Agreed about a read-lock-only option when calling a put though, this will helpimprove concurrency a good deal more.

Curious, how do the likes of EHCache, etc. deal with this? I don't believe they have 2 different put() methods...

[otaylor]

Wanted to pick up this old thread and add a few observations I've made:

EHCache and similar are working on an entirely different code path ... the CacheConcurrencyStrategy is ReadWriteCache rather than TransactionalCache; ReadWriteCache is a pretty complex scheme, but the basic idea of it is that it stores lock objects in the cache that wrap the actual cached objects and keeps lock state and timestamps in the lock objects.

AFAIK the two TreeCache based caches are the only users of TransactionalCache.

Yes, the fail-fast variant of put is used currently. Issues that I'm aware of with it:

- It doesn't work with optimistic locking (just broken? doesn't make senes? not sure)
- It doesn't replicate quite properly (the "fail-silently" option isn't replicated)
- It isn't handled for INVALIDATE_SYNC at all right (The 0-timeout isn't propagate)
- It's not 100% correct

To expand on the last, the problem is that you can have:

a) get() returns null
b) new data or delete comimtted in database and cache
c) put() overwrites the newer data

I'd argue that get() needs to establish a hard read lock (even when the node isn't there, even if the isolation mode is READ_COMMITTED) that is upgraded to a write lock on put() and then released/downgraded afterwards. (Doing the get()/put() pair in their own transaction might make sense, just as the fail-fast put is done in its own transaction now)

This would likely require some Hibernate <=> cache changes to make sure that a failed get() is always followed by another call (a put() on success, but the item might not have been in the database, or there could have been other failures.)

The above is of course somewhat similar to what the cache loader infrastructure does, but wedging into that would be more invasive on the Hibernate side.

Anyways, I don't want to go into a lot more detail here, but just wanted to point out that dealing only with the put() and not considering the get()/put() cycle has some issues.

[manik]

putFailFast() is a kludge that was introduced in earlier versions of JBC, and is deprecated.

The correct strategy for achieving this behaviour - regardless of node locking scheme - is to set an Option - FAIL_SILENTLY.

[otaylor]

I was just using "fail-fast" as a shorthand for the whole set of behaviors that are used now:

- Lock temporarily, not part of the current transaction
- Time out immediately on lock failure
- Don't throw an exception on lock failure

I'm not sure I buy that these are all really independent, especially when have additional special lock handling as discussed here. But I didn't mean to imply that putFailFast was actually used.

[otaylor]

Hmmm, dug around in this some more and it turns out that (the non-optimistic) Hibernate TreeCache glue *is* still literally using putFailFast, even in latest CVS. Looking at the details of the behavior bundle, I'm not sure it could be switched over without some adiditional work.

Isolated from parent transaction:
putFailFast: Doesn't provide, Hibernate suspends its transaction itself before calling into the TreeCache. (There's a bug here, the suspended transaction leaks onto the invocation context used for putFailFast, so putFailFast creates its locks owned by the suspended transaction!)
FAIL_SILENTLY: Provides automatically

Replicates asynchronously even if the cache is REPL_SYNC
putFailFast: Provides (hacked into the ReplicationInterceptor)
FAIL_SILENTLY: Doesn't provide. Maybe could be a separate option?

Lock failure immediate without waiting for a lock timeout:
putFailFast: Provides
FAIL_SILENTLY: Doesn't provide (??? Am I missing something?)

Lock failure ignored:
putFailFast: Doesn't provide; for local failure, caller can catch TimeoutException. failure on (async) replication is logged at level ERROR.
FAIL_SILENTLY: Provides, sort of. Except for the one particular case of JBCACHE-767 (now) silent isn't very silent... it logs at level INFO.

For the short term I'm going to see if I can come up with fixes for the two putFailFast bugs noted above (leak of locks onto suspended transaction, ERROR logging on replication).

[otaylor]

JIRA references for the two bugs:

JBCACHE-785: InvocationContext and suspended transactions
http://jira.jboss.com/jira/browse/JBCACHE-785
(I didn't come up with an easy fix. Hibernate workaround patch included)

JBCACHE-786: errors from "OK" putFailFast replications
http://jira.jboss.com/jira/browse/JBCACHE-786
(Patch included)

[manik]

Thanks for this stuff, Owen - will look into it soon.

[brian.stansberry]

There was a brief off-line discussion on the topic of adding the ability to specify lock-types externally (maybe via Options) such that Hibernate could simply request that "this put only acquire a read lock". Here's the key point:

"Manik Surtani" wrote:
I don't think specifying lock types will work, since this is only symbolic. (i.e., with optimistic locking, locking is completely different)

Perhaps what we need, literally, is completely separate behaviour for put() when the FAIL_SILENTLY option is used -which leads me to think that this shouldn't be an option at all, but a separate put() method - perhaps putFailFast() again, but implemented quite differently, involving all the specific needs of this put(), including:

* suspended txs
* minimal locking
* 0 timeouts
* async replication

+1 on a separate method. I'm the one who raised the original Option idea, but I think now that was wrongheaded. The term "Hibernate 2nd level cache" makes the use case sound specialized, but really it's one of the basic use cases for JBC -- read from database, then write to cache to avoid future reads. Programming for that use case needs to be dead simple; having to deal with Options is too much.