BerkeleyDB Deadlock Exception from JBoss Cache ...| JBoss.org Content Archive (Read Only)

15. Re: BerkeleyDB Deadlock Exception from JBoss Cache Loader: B

niuxuetao_fiserv Jul 20, 2009 7:04 AM (in response to niuxuetao_fiserv)

Thanks Galder!

I doubt managing the number of nodes can greatly increase the difficulty of our implementation or deployment when we use Pojo cache. Or perhaps it is even not doable to dynamically manage the node numbers?

I am wondering if there is a jboss product (perhaps infini-span) that overcomes the lock per FQN thing or does the dynamic node management transparently.

Another extreme way is that we let each node contain one-entry-map and play with the entire tree/FQN-set in a concurrent way. Please confirm if this is feasible performance wise when using Pojo cache.

Thanks,

Xuetao

16. Re: BerkeleyDB Deadlock Exception from JBoss Cache Loader: B

niuxuetao_fiserv Jul 21, 2009 7:42 AM (in response to niuxuetao_fiserv)

I just realized that since I am using Pojo cache, I am not sure how Pojo cache creates the FQN when I pass a string id to the PojoCache.attach(String id,Object obj).

Suppose I pass id as "X/Y" what FQN will be really created?

What will be the performance impact if I have 10-20 variations of "X" and for each X I have 1000,000 variations of "Y"?

I thought Pojo cache would use X as FQN and Y as the map key when it invokes the core cache, which could be very wrong...

How would I deal with the segmentation of the id strings when I am using Pojo cache for good performance?

17. Re: BerkeleyDB Deadlock Exception from JBoss Cache Loader: B

niuxuetao_fiserv Jul 21, 2009 7:57 AM (in response to niuxuetao_fiserv)

Now I have a very serious performance problem with the way (the "X/Y" way) I compose the string id when using Pojo cache.

I have one PojoCache instance globally shared by 20 threads in websphere, but those 20 threads are mostly in the "waiting" state by what I see with JProfiler, the hotspots are:
- 6.2% - 30,415 s - 9,479 hot spot inv. org.jboss.cache.util.concurrent.locks.OwnableReentrantLock.tryLock
- 2.0% - 10,057 s - 5,065 hot spot inv. org.jboss.cache.mvcc.MVCCNodeHelper.acquireLock (line: 259)

While the threads are waiting, the CPU usage is nearly zero (there are some daemon threads running periodically in websphere).

It is not a real deadlock because all processing eventually finished without. What I don't understand is that why in majority of the time those threads are waiting for each other but no one is really busy?

No matter how I compose the key, I would expect at least one thread is busy at a certain moment.

18. Re: BerkeleyDB Deadlock Exception from JBoss Cache Loader: B

niuxuetao_fiserv Jul 21, 2009 8:00 AM (in response to niuxuetao_fiserv)

I mean the healthy concurrency situation I would expect to see is that at least one thread is busy at ANY moment.

19. Re: BerkeleyDB Deadlock Exception from JBoss Cache Loader: B

niuxuetao_fiserv Jul 21, 2009 11:04 AM (in response to niuxuetao_fiserv)

The stack trace of the waiting thread is below:

org.jboss.cache.util.concurrent.locks.OwnableReentrantLock.tryLock(long, java.util.concurrent.TimeUnit)
java.util.concurrent.locks.Lock.tryLock(long, java.util.concurrent.TimeUnit)
org.jboss.cache.util.concurrent.locks.AbstractSharedLockContainer.acquireLock(java.lang.Object, long, java.util.concurrent.TimeUnit)
org.jboss.cache.lock.MVCCLockManager.lockAndRecord(org.jboss.cache.Fqn, org.jboss.cache.lock.LockType, org.jboss.cache.InvocationContext)
org.jboss.cache.mvcc.MVCCNodeHelper.acquireLock(org.jboss.cache.InvocationContext, org.jboss.cache.Fqn)
org.jboss.cache.mvcc.MVCCNodeHelper.wrapNodeForWriting(org.jboss.cache.InvocationContext, org.jboss.cache.Fqn, boolean, boolean, boolean, boolean, boolean, boolean)
org.jboss.cache.mvcc.MVCCNodeHelper.wrapNodeForWriting(org.jboss.cache.InvocationContext, org.jboss.cache.Fqn, boolean, boolean, boolean, boolean, boolean)
org.jboss.cache.interceptors.CacheLoaderInterceptor.loadIfNeeded(org.jboss.cache.InvocationContext, org.jboss.cache.Fqn, java.lang.Object, boolean, boolean, boolean, boolean, boolean, boolean, boolean)
org.jboss.cache.interceptors.CacheLoaderInterceptor.visitGetKeyValueCommand(org.jboss.cache.InvocationContext, org.jboss.cache.commands.read.GetKeyValueCommand)
org.jboss.cache.commands.read.GetKeyValueCommand.acceptVisitor(org.jboss.cache.InvocationContext, org.jboss.cache.commands.Visitor)
org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(org.jboss.cache.InvocationContext, org.jboss.cache.commands.VisitableCommand)
org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(org.jboss.cache.InvocationContext, org.jboss.cache.commands.VisitableCommand)
org.jboss.cache.commands.AbstractVisitor.visitGetKeyValueCommand(org.jboss.cache.InvocationContext, org.jboss.cache.commands.read.GetKeyValueCommand)
org.jboss.cache.commands.read.GetKeyValueCommand.acceptVisitor(org.jboss.cache.InvocationContext, org.jboss.cache.commands.Visitor)
org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(org.jboss.cache.InvocationContext, org.jboss.cache.commands.VisitableCommand)
org.jboss.cache.interceptors.TxInterceptor.attachGtxAndPassUpChain(org.jboss.cache.InvocationContext, org.jboss.cache.commands.VisitableCommand)
org.jboss.cache.interceptors.TxInterceptor.handleDefault(org.jboss.cache.InvocationContext, org.jboss.cache.commands.VisitableCommand)
org.jboss.cache.commands.AbstractVisitor.visitGetKeyValueCommand(org.jboss.cache.InvocationContext, org.jboss.cache.commands.read.GetKeyValueCommand)
org.jboss.cache.commands.read.GetKeyValueCommand.acceptVisitor(org.jboss.cache.InvocationContext, org.jboss.cache.commands.Visitor)
org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(org.jboss.cache.InvocationContext, org.jboss.cache.commands.VisitableCommand)
org.jboss.cache.interceptors.InvocationContextInterceptor.handleAll(org.jboss.cache.InvocationContext, org.jboss.cache.commands.VisitableCommand, org.jboss.cache.transaction.GlobalTransaction, boolean)
org.jboss.cache.interceptors.InvocationContextInterceptor.handleDefault(org.jboss.cache.InvocationContext, org.jboss.cache.commands.VisitableCommand)
org.jboss.cache.commands.AbstractVisitor.visitGetKeyValueCommand(org.jboss.cache.InvocationContext, org.jboss.cache.commands.read.GetKeyValueCommand)
org.jboss.cache.commands.read.GetKeyValueCommand.acceptVisitor(org.jboss.cache.InvocationContext, org.jboss.cache.commands.Visitor)
org.jboss.cache.interceptors.InterceptorChain.invoke(org.jboss.cache.InvocationContext, org.jboss.cache.commands.VisitableCommand)
org.jboss.cache.invocation.CacheInvocationDelegate.get(org.jboss.cache.Fqn, java.lang.Object)
org.jboss.cache.pojo.impl.InternalHelper.get(org.jboss.cache.Fqn, java.lang.Object, boolean)
org.jboss.cache.pojo.impl.InternalHelper.getPojoReference(org.jboss.cache.Fqn, java.lang.String)
org.jboss.cache.pojo.impl.InternalHelper.getPojo(org.jboss.cache.Fqn, java.lang.String)
org.jboss.cache.pojo.impl.PojoCacheDelegate.getObject(org.jboss.cache.Fqn, java.lang.String, java.lang.Object)
org.jboss.cache.pojo.impl.PojoCacheImpl.find(org.jboss.cache.Fqn, java.lang.String, java.lang.Object)
org.jboss.cache.pojo.impl.PojoCacheImpl.find(org.jboss.cache.Fqn)
org.jboss.cache.pojo.impl.PojoCacheImpl.find(java.lang.String)
com.fiserv.detection.eventcache.jboss.JBossPojoCacheStorage.get(java.lang.String)

20. Re: BerkeleyDB Deadlock Exception from JBoss Cache Loader: B

galder.zamarreno Jul 22, 2009 3:35 AM (in response to niuxuetao_fiserv)

"niuxuetao_fiserv" wrote:
I just realized that since I am using Pojo cache, I am not sure how Pojo cache creates the FQN when I pass a string id to the PojoCache.attach(String id,Object obj).

Suppose I pass id as "X/Y" what FQN will be really created?

What will be the performance impact if I have 10-20 variations of "X" and for each X I have 1000,000 variations of "Y"?

I thought Pojo cache would use X as FQN and Y as the map key when it invokes the core cache, which could be very wrong...

How would I deal with the segmentation of the id strings when I am using Pojo cache for good performance?

No, the id is purely the FQN. It pays off to use several layers of FQN if you have a lot of objects so that these objects can be acted on concurrently. So, I'd suggest something like, this were Y is used as a bucket layer: X/Y/Z. So, you could have 10-20 diff X, 1000 Y buckets and within each Y, 1000 Z nodes.

21. Re: BerkeleyDB Deadlock Exception from JBoss Cache Loader: B

galder.zamarreno Jul 22, 2009 3:54 AM (in response to niuxuetao_fiserv)

With regards to the performance issue, please send me several thread dumps taken with 30s-1m difference each using the kill -3 or Ctrl+Break method explained in http://www.jboss.org/community/wiki/StackTrace and enable garbage collection logging http://www.jboss.org/community/wiki/TuneVMGarbageCollection

My suggestions here would be to try to spread the pojos under different FQNs making 3 or 4 layer structure if you have a lot of pojos. Please also look at reducing the number of detach() operations called, or attach() where you're attaching a Pojo to an existing Fqn.

22. Re: BerkeleyDB Deadlock Exception from JBoss Cache Loader: B

niuxuetao_fiserv Jul 22, 2009 9:16 AM (in response to niuxuetao_fiserv)

Thanks for your suggestions!

For this I have a question:

No, the id is purely the FQN.

It sounds like one FQN is associated to only one pojo that is attached with the string id. There cannot be multiple pojos (we do not count the nested inner pojos of one attached pojo) under the same FQN. Is my understanding correct?

If so, I am not doing (I actually did before) attach() on an existing FQN. Instead, I always use find() first to see if the FQN is already associated with a pojo, if found, directly update it or remove it; if not found, call attach(). (pity that there is no attachIfAbsent() from pojo cache api in version 3.0.0)

By the current way I still see the performance problem, which seems nothing to do with the layers of FQN since all threads are waiting.

I have sent the thread dumps, with which you can probably pinpoint something.

Many thanks!

Xuetao