CacheLoaderInterceptor Bottleneck?

[tallpsmith]

If our production server requires a restart in peak times, we have noticed incredibly slow throughput in the early minutes after.

Refering to my comments and thread dumps here:

http://jira.jboss.com/jira/browse/JBCACHE-512

I've been looking at the CacheLoaderInterceptor class, and I'm really not sure why there is a synchronized(this) blok in the invoke method with the comment:

/* On the way in: load elements into cache from the CacheLoader if not yet in the cache. We need to synchronize
this so only 1 thread attempts to load a given element */

Well.. err.. if the loading of an element isn't fast (and the reason it's being stored in the cache in the first place is that the lookup is not cheap) then this is going to serialize access to the instance of CacheLoaderInterceptor surely... We'r seeing dozens and dozens of threads blocked waiting for the node to load.

See the thread dumps attached to the JIRA issue.

This can't be healthy.. ? Could someone explain why the need to synch on that instance? Isn't the locks on the fqn itself enough?

[tallpsmith]

Actually, this is all encapsulated in:

http://jira.jboss.com/jira/browse/JBCACHE-513

See my test case attached to that issue.

[manik]

Hi, thanks for your test case. It proves this inefficiency quite well.

I'm working on a proper fix involving locking on a per-fqn level based on method and isolation level, but for now I have a fix in CVS which does improve efficiency considerably but is not the final fix. If this "halfway fix" works and passes regression testing and the rest, I'd be keen to release it in 1.3.0 rather than hold out for 1.4.0 for a proper fix.

I'd encourage you to grab the latest JBossCache off CVS HEAD, build it and give it a try, see what you think.

Cheers,
Manik

[tallpsmith]

Manik,

Thanks for you reply, glad the test case helped (I wrote in the car on the way home! :) - my wife was driving..)

I'll grab CVS Head and check out the changes. At this stage, _any_ efficiency gains in the 1.3.0 release would be most sincerely appreciated.

If it helps, I could help wrlte some more test cases?

[manik]

Just as well you didn't write the test case while *you* were driving. :)

The test case is fine so far, but if you would like to contribute further for more performance benchmarking and some of the major improvements I have planned for the 1.4.0 timeframe, do let me know - I could always use an extra pair of capable hands.

[belaban]

Just as a waning: if you try implementing locking per FQN, you will end up creating a tree-like structure again ! IMO we should better implement this in the cache itself, and use multiple root for example to bypass this problem.
Or - if you absolutely have to do it - extract this functionality into its own Locking Table, I created a JIRA issue some time ago. This LT could be used for
- pessimistic locking (per FQN)
- briefly locking the tree in OL