BerkeleyDB Deadlock Exception from JBoss Cache Loader: Bdbje
niuxuetao_fiserv Jun 20, 2009 5:23 AMHi,
I am using Jboss Pojo cache on top of Jboss cache (3.0.0GA) managing rather big object trees of collections. And I am randomly getting com.sleepycat.je.DeadlockException(s) when using the BdbjeCacheLoader as cache loader.
There are 10 threads running under a single machine (JSE environment), each does both reading and writing. Neither reading nor writing (as I believe) should take up to 20 seconds as a transactional batch on my hardware, since the objects are not huge (each object is a list of other object that include an array of maps of basic types).
If I set the lockAcquisitionTimeout to smaller value, it happens more frequently. But even I set the value to a very big one, like 20000, it still happens with less frequency, and it blocks for 20 seconds before this exception is thrown.
Due to these posts, I guess it might have something to do with mvcc. (I am using je-3.3.82), but really do not hope so, since mvcc is one of the points I like jboss cache.
http://forums.oracle.com/forums/thread.jspa?threadID=866063&tstart=60
http://forums.oracle.com/forums/thread.jspa?threadID=616358&tstart=30
Below is the jboss cache configuration and the exception stack trace. Thanks in advance for your help!
<?xml version="1.0" encoding="UTF-8"?> <jbosscache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:jboss:jbosscache-core:config:3.0"> <locking isolationLevel="READ_COMMITTED" lockParentForChildInsertRemove="false" lockAcquisitionTimeout="5000" nodeLockingScheme="mvcc" writeSkewCheck="false" concurrencyLevel="500" /> <transaction transactionManagerLookupClass="org.jboss.cache.transaction.GenericTransactionManagerLookup" syncRollbackPhase="false" syncCommitPhase="false" /> <jmxStatistics enabled="false" /> <startup regionsInactiveOnStartup="true" /> <shutdown hookBehavior="DEFAULT" /> <listeners asyncPoolSize="1" asyncQueueSize="100000" /> <invocationBatching enabled="false" /> <eviction wakeUpInterval="1000"> <!-- Cache wide default --> <region name="/_default_" algorithmClass="org.jboss.cache.eviction.LRUAlgorithm"> <property name="maxNodes">10000</property> <property name="timeToLiveSeconds">300</property> </region> </eviction> <loaders passivation="false" shared="true"> <preload> <node fqn="/" /> </preload> <loader class="org.jboss.cache.loader.bdbje.BdbjeCacheLoader" async="false" fetchPersistentState="false" ignoreModifications="false" purgeOnStartup="false"> <properties> cache.jdbc.table.name=Cached_Events cache.jdbc.table.create=false cache.jdbc.table.drop=false cache.jdbc.table.primarykey=jbosscache_pk cache.jdbc.fqn.column=fqn cache.jdbc.fqn.type=varchar2(255) cache.jdbc.node.column=node cache.jdbc.node.type=blob cache.jdbc.parent.column=parent cache.jdbc.sql-concat=concat(1,2) cache.jdbc.connection.factory=org.jboss.cache.loader.C3p0ConnectionFactory c3p0.maxPoolSize=20 c3p0.checkoutTimeout=5000 location=tmp/jbosscache </properties> </loader> </loaders> </jbosscache>
com.sleepycat.je.DeadlockException: (JE 3.3.82) Lock expired. Locker 6761395 -1_Detection2_ThreadLocker: waited for lock on database=JBossCache-Cluster LockAddr:23471145 node=1785 type=READ grant=WAIT_NEW timeoutMillis=20000 startTime=1245426236094 endTime=1245426256097 [junit] Owners: [<LockInfo locker="12706060 346_Detection1_Txn" type="WRITE"/>] [junit] Waiters: [<LockInfo locker="24478649 347_Detection4_Txn" type="WRITE"/>] [junit] [junit] at com.sleepycat.je.txn.LockManager.makeTimeoutMsgInternal(LockManager.java:469) [junit] at com.sleepycat.je.txn.SyncedLockManager.makeTimeoutMsg(SyncedLockManager.java:77) [junit] at com.sleepycat.je.txn.LockManager.lock(LockManager.java:278) [junit] at com.sleepycat.je.txn.BasicLocker.lockInternal(BasicLocker.java:167) [junit] at com.sleepycat.je.txn.Locker.lock(Locker.java:360) [junit] at com.sleepycat.je.dbi.CursorImpl.lockLNDeletedAllowed(CursorImpl.java:2516) [junit] at com.sleepycat.je.dbi.CursorImpl.lockLN(CursorImpl.java:2438) [junit] at com.sleepycat.je.dbi.CursorImpl.fetchCurrent(CursorImpl.java:2368) [junit] at com.sleepycat.je.dbi.CursorImpl.getCurrentAlreadyLatched(CursorImpl.java:1427) [junit] at com.sleepycat.je.dbi.CursorImpl.getNextWithKeyChangeStatus(CursorImpl.java:1573) [junit] at com.sleepycat.je.dbi.CursorImpl.getNext(CursorImpl.java:1499) [junit] at com.sleepycat.je.dbi.CursorImpl.getNextNoDup(CursorImpl.java:1688) [junit] at com.sleepycat.je.Cursor.searchInternal(Cursor.java:1936) [junit] at com.sleepycat.je.Cursor.searchAllowPhantoms(Cursor.java:1748) [junit] at com.sleepycat.je.Cursor.search(Cursor.java:1615) [junit] at com.sleepycat.je.Cursor.getSearchKeyRange(Cursor.java:1117) [junit] at org.jboss.cache.loader.bdbje.BdbjeCacheLoader.getChildrenNames(BdbjeCacheLoader.java:391) [junit] at org.jboss.cache.loader.CacheLoaderManager.preload(CacheLoaderManager.java:371)