deadlock in put to infinispan?
omerzohar Mar 23, 2011 10:58 AMso it goes like this : i have a lot of MDBs running in a JBOSS AS 6 + Infinispan configuration heavly using the cache, puting and geting object. at one poing while running my server i notcied it doesn't do much.
I noticed the Average write time is 4.5 seconds( ! )
Ii did a thread dump and basically all threads are stuck on the PUT method of infinispan
this is the stacktrace example of one of the thread (they're all identical more or less):
Thread: Thread-16 (group:HornetQ-client-global-threads-31731945) : priority:5, demon:true, threadId:174, threadState:WAITING - waiting on <0x69b10> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114) java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807) org.infinispan.util.concurrent.locks.StripedLock.acquireLock(StripedLock.java:94) org.infinispan.loaders.LockSupportCacheStore.lockForWriting(LockSupportCacheStore.java:65) org.infinispan.loaders.LockSupportCacheStore.store(LockSupportCacheStore.java:149) org.infinispan.interceptors.DistCacheStoreInterceptor.visitPutKeyValueCommand(DistCacheStoreInterceptor.java:81) org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76) org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) org.infinispan.interceptors.CacheLoaderInterceptor.visitPutKeyValueCommand(CacheLoaderInterceptor.java:81) org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76) org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:58) org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76) org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) org.infinispan.interceptors.MarshalledValueInterceptor.visitPutKeyValueCommand(MarshalledValueInterceptor.java:125) org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76) org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:182) org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:130) org.infinispan.interceptors.DistTxInterceptor.visitPutKeyValueCommand(DistTxInterceptor.java:76) org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76) org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:113) org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76) org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:87) org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:58) org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:58) org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76) org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:76) org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:58) org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76) org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273) org.infinispan.CacheDelegate.put(CacheDelegate.java:444) org.infinispan.CacheSupport.put(CacheSupport.java:28)
And this is my Cache config:
<!-- blackbox application cache bean cache definitions --> <infinispan-config name="blackBox" jndi-name="java:CacheManager/blackbox"> <alias>blackBox-cache</alias> <infinispan xmlns="urn:infinispan:config:4.2"> <global> <transport clusterName="${jboss.partition.name:DefaultPartition}-blackBox" distributedSyncTimeout="17500"> <properties> <property name="stack" value="${jboss.default.jgroups.stack:udp}"/> </properties> </transport> <globalJmxStatistics enabled="true"/> <shutdown hookBehavior="DONT_REGISTER"/> </global> <default> <locking isolationLevel="REPEATABLE_READ" lockAcquisitionTimeout="15000" useLockStriping="false" concurrencyLevel="1000"/> <jmxStatistics enabled="true"/> <lazyDeserialization enabled="true"/> <invocationBatching enabled="true"/> <eviction wakeUpInterval="5000" strategy="LRU"/> </default> <!-- DATA PERSISTANT CACHE --> <namedCache name="persistant"> <clustering mode="distribution"> <stateRetrieval timeout="60000" fetchInMemoryState="false"/> <sync/> <hash numOwners="2"/> <l1 enabled="true"/> </clustering> <loaders passivation="false" shared="true" preload="true"> <loader class="org.infinispan.loaders.file.FileCacheStore" fetchPersistentState="false" purgeOnStartup="false"> <properties> <property name="location" value="${jboss.server.data.dir}${/}blackbox.cache"/> </properties> </loader> </loaders> <!--eviction wakeUpInterval="600000" maxEntries="1024" strategy="LIRS"/> <expiration lifespan="86400000" maxIdle="-1"/--> <eviction wakeUpInterval="60000" maxEntries="16384" strategy="LIRS"/> <expiration lifespan="14400000" maxIdle="-1"/> </namedCache>
Anyone has an advice how to resolve this? am i using infinispan wrong or something?