Various errors showing up when load testing

[memoryerror]

When we run our application with multiple users, various problems start showing up in the logs.

In particular:

org.jboss.cache.lock.TimeoutException: read lock for [...] could not be acquired by GlobalTransaction::90620 after 15000 ms. Locks: Read lock owners: [] Write lock owner: GlobalTransaction::90595

What's odd is there don't seem to be any read locks when this happens, according to the error message.

Does anyone recognize this error? If so, what should we be looking at to fix the problem?

(We're using 1.4.1.SP9.)

[manik]

If there is a writer, no one can acquire a read lock, as the log line suggests. I'm guessing you are using REPEATABLE_READ?

[memoryerror]

I'm sorry I didn't respond sooner--I had registered to watch this topic for replies, but never got a notification!

It's not clear that we can use a different read mode here, because the cache is being written to due to a cache fault--a reader got a null, so the reader fetches from the database and does the put. Meanwhile all other readers should block, because if they don't and are allowed to read, they'll read nulls as well, and also fetch from the database, and also attempt to put, and I'll wind up with multiple writers getting in each others' way.

The puzzling thing is that the put to the cache should be virtually instantaneous, so I don't see where the 15+ second delay is coming from. It's true that the system is under load, but it's not under tremendous load.

[manik]

For the use-case of caching data which could be read from a database, use putForExternalRead(). Have a look at the Javadocs there (this is a 2.x and 3.x API), this is how Hibernate and other persistence mechanisms use the cache to store data read from elsewhere.

[memoryerror]

Based on an earlier thread on this forum that I started, and their response from JBoss (which I think was from you, Manik!) asking how to avoid notification on put, we call setCacheModeLocal(true) for puts of data read from the database as opposed to puts of new data. Will that not accomplish what we want? Also, how would calling putForExternalRead solve the TimeoutException?

[memoryerror]

Here's the other thread: http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4173985.

[memoryerror]

If we use putForExternalRead, do we no longer need to call setCacheModeLocal?

[memoryerror]

If the put fails, the next call to get will fault and go to the database, and the next, and so forth, with each of them doing a fetch of data and attempting a put. So now instead of blocking reads behind one write (gets behind one put), we'll have write contention due to multiple puts, which is what we were trying to avoid.

[memoryerror]

The comments in the source for putForExternalRead say See JBCACHE-848 for details around this feature., but that link is dead.

[manik]

"jshowalter" wrote:
The comments in the source for putForExternalRead say See <a href="http://jira.jboss.com/jira/browse/JBCACHE-848">JBCACHE-848</a> for details around this feature., but that link is dead.

The link works for me ... it goes here:

https://jira.jboss.org/jira/browse/JBCACHE-848

[manik]

"jshowalter" wrote:
Based on an earlier thread on this forum that I started, and their response from JBoss (which I think was from you, Manik!) asking how to avoid notification on put, we call setCacheModeLocal(true) for puts of data read from the database as opposed to puts of new data. Will that not accomplish what we want? Also, how would calling putForExternalRead solve the TimeoutException?

pFER implicitly sets your timeout to 0 so it won't wait for locks. It it cannot acquire the lock *immediately*, it won't fail either - it will just not do anything. This is OK since the only time it won't be able to get a write lock is if some other thread is doing the same thing (reading form DB and writing to cache) so this is acceptable.

[memoryerror]

Yesterday that link didn't work, but today it does. I've read the discussion of pFER. It looks like it was designed to do exactly what we need. Thank you very much for your help!

[memoryerror]

Hoo boy--you guys have been busy making 2.0 non backward compatible with 1.*.

For example:

1) Renamed TreeCache to Cache, or in some other way gotten rid of TreeCache.

2) Changed Cache to Cache<K,V>.

3) Changed Fqn to Fqn<K,V>.

Is there updated documentation?

[memoryerror]

(I know there are updated Javadocs--by updated documentation, I mean are there 2.0 versions of:

http://www.jboss.org/file-access/default/members/jbosscache/freezone/docs/1.4.1.SP4/TreeCache/en/html/index.html

http://www.jboss.org/jbosscache/docs/1.4.0/faq/en/html/index.html

?)

[memoryerror]

Can we even use the 2.0 version of the cache in Jboss 4.3.CP02?

Would it be simpler for us to take the source code and implement our own putForExternalRead locally, or did you make fundamental changes to the implementation in order to add this method? If it's just a matter of setting the timeout, etc., we could replicate what you did without buying into the rest of the changes.