Cache lock (JbossCache 1.4.1.SP3)
sancheski Nov 26, 2007 12:43 PMHi all
We are having some kind of strange locks that comes from our cache that we have in production. This problem have come up from two different situations:
The first one is during a scheduler execution that just evicts a node at certain time.
The second one is during normal applications execution.
This second situation is quite confused to us. Stack trace below:
"http-170.32.21.12-8080-250" daemon prio=10 tid=6000000007078c50 nid=43815 lwp_id=9087433 in Object.wait() [9fffffffd84c0000..9ffff fffd84c0c40] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:474) at org.jboss.cache.interceptors.BaseCacheLoaderInterceptor.obtainLoaderLock(BaseCacheLoaderInterceptor.java:49) - locked <9ffffffee2c31308> (a org.jboss.cache.interceptors.CacheStoreInterceptor) at org.jboss.cache.interceptors.BaseCacheLoaderInterceptor.obtainLoaderLocks(BaseCacheLoaderInterceptor.java:91) at org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:92) at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68) at org.jboss.cache.interceptors.TxInterceptor.handleCommitRollback(TxInterceptor.java:719) at org.jboss.cache.interceptors.TxInterceptor.runCommitPhase(TxInterceptor.java:761) at org.jboss.cache.interceptors.TxInterceptor$RemoteSynchronizationHandler.afterCompletion(TxInterceptor.java:1076) at org.jboss.cache.interceptors.TxInterceptor$LocalSynchronizationHandler.afterCompletion(TxInterceptor.java:1175) at org.jboss.cache.interceptors.OrderedSynchronizationHandler.afterCompletion(OrderedSynchronizationHandler.java:83) at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:136) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:342) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:95) at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:177) at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1256) at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:135) at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:87) at org.jboss.tm.usertx.client.ServerVMClientUserTransaction.commit(ServerVMClientUserTransaction.java:140)
This cache lock leads to a 8080 port blocking, that means no service availability.
Our cache configuration is as follows:
<server> <classpath codebase="./lib" archives="jboss-cache.jar, jgroups.jar"/> <!-- ==================================================================== --> <!-- Defines TreeCache configuration --> <!-- ==================================================================== --> <mbean code="org.jboss.cache.TreeCache" name="jboss.cache:service=FooCache"> <depends>jboss:service=Naming</depends> <depends>jboss:service=TransactionManager</depends> <attribute name="TransactionManagerLookupClass">org.jboss.cache.JBossTransactionManagerLookup</attribute> <!-- Isolation level : SERIALIZABLE REPEATABLE_READ (default) READ_COMMITTED READ_UNCOMMITTED NONE --> <attribute name="IsolationLevel">NONE</attribute> <!-- Valid modes are LOCAL REPL_ASYNC REPL_SYNC INVALIDATION_ASYNC INVALIDATION_SYNC --> <attribute name="CacheMode">REPL_SYNC</attribute> <!-- Just used for async repl: use a replication queue --> <attribute name="UseReplQueue">false</attribute> <!-- Replication interval for replication queue (in ms) --> <attribute name="ReplQueueInterval">0</attribute> <!-- Max number of elements which trigger replication --> <attribute name="ReplQueueMaxElements">0</attribute> <!-- Name of cluster. Needs to be the same for all clusters, in order to find each other --> <attribute name="ClusterName">FooCache-Cluster</attribute> <!-- JGroups protocol stack properties. Can also be a URL, e.g. file:/home/bela/default.xml <attribute name="ClusterProperties"></attribute> --> <attribute name="ClusterConfig"> <config> <!-- UDP: if you have a multihomed machine, set the bind_addr attribute to the appropriate NIC IP address, e.g bind_addr="192.168.0.2" --> <!-- UDP: On Windows machines, because of the media sense feature being broken with multicast (even after disabling media sense) set the loopback attribute to true --> <UDP mcast_addr="${jboss.cache.FooCache.addr:228.1.2.3}" mcast_port="${jboss.cache.FooCache.port:48866}" ip_ttl="64" ip_mcast="true" mcast_send_buf_size="150000" mcast_recv_buf_size="80000" ucast_send_buf_size="150000" ucast_recv_buf_size="80000" loopback="false"/> <PING timeout="2000" num_initial_members="3" up_thread="false" down_thread="false"/> <MERGE2 min_interval="10000" max_interval="20000"/> <!-- <FD shun="true" up_thread="true" down_thread="true" />--> <FD_SOCK/> <VERIFY_SUSPECT timeout="1500" up_thread="false" down_thread="false"/> <pbcast.NAKACK gc_lag="50" retransmit_timeout="600,1200,2400,4800" max_xmit_size="8192" up_thread="false" down_thread="false"/> <UNICAST timeout="600,1200,2400" window_size="100" min_threshold="10" down_thread="false"/> <pbcast.STABLE desired_avg_gossip="20000" up_thread="false" down_thread="false"/> <FRAG frag_size="8192" down_thread="false" up_thread="false"/> <pbcast.GMS join_timeout="5000" join_retry_timeout="2000" shun="true" print_local_addr="true"/> <pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/> </config> </attribute> <!-- Whether or not to fetch state on joining a cluster NOTE this used to be called FetchStateOnStartup and has been renamed to be more descriptive. --> <attribute name="FetchInMemoryState">true</attribute> <!-- The max amount of time (in milliseconds) we wait until the initial state (ie. the contents of the cache) are retrieved from existing members in a clustered environment --> <attribute name="InitialStateRetrievalTimeout">15000</attribute> <!-- Number of milliseconds to wait until all responses for a synchronous call have been received. --> <attribute name="SyncReplTimeout">15000</attribute> <!-- Max number of milliseconds to wait for a lock acquisition --> <attribute name="LockAcquisitionTimeout">10000</attribute> <!-- Name of the eviction policy class. --> <attribute name="EvictionPolicyClass">org.jboss.cache.eviction.LRUPolicy</attribute> <!-- Specific eviction policy configurations. This is LRU --> <attribute name="EvictionPolicyConfig"> <config> <!-- This attribute will be share by all eviction policies --> <attribute name="wakeUpIntervalSeconds">5</attribute> <!-- Cache wide default 86400--> <region name="/_default_"> <attribute name="maxNodes">500000</attribute> <attribute name="timeToLiveSeconds">86400</attribute> </region> </config> </attribute> <!-- Indicate whether to use region based marshalling or not. Set this to true if you are running under a scoped class loader, e.g., inside an application server. Default is "false". --> <attribute name="UseRegionBasedMarshalling">true</attribute> <attribute name="InactiveOnStartup">true</attribute> <attribute name="CacheLoaderConfiguration"> <config> <passivation>false</passivation> <preload>/</preload> <shared>true</shared> <!-- Own implementation of a ClusteredCacheLoader that allows act/deact of regions --> <cacheloader> <class> com.foo.bar.loader.ExtendedClusteredCacheLoader </class> <properties>timeout=1000</properties> <async>true</async> <fetchPersistentState>false</fetchPersistentState> <ignoreModifications>false</ignoreModifications> </cacheloader> </config> </attribute> </mbean> <!-- Uncomment to get a graphical view of the TreeCache MBean above --> <!-- <mbean code="org.jboss.cache.TreeCacheView" name="jboss.cache:service=TreeCacheView">--> <!-- <depends>jboss.cache:service=TreeCache</depends>--> <!-- <attribute name="CacheService">jboss.cache:service=TreeCache</attribute>--> <!-- </mbean>--> </server>
Hope someone could help us.
Thanks in advance!