org.jboss.cache.lock.TimeoutException

[manishobhatia]

Hi,
We are using JBoss Cache 1.3.0 with JBoss AS 4.0.3SP1. We face this lock timeout exception when there is high load on the cache.

Any help on this issue will be highly appreciated

This is what the exception looks like
Caused by: java.lang.RuntimeException:
at org.jboss.cache.interceptors.TxInterceptor$LocalSynchronizationHandler.beforeCompletion(TxInterceptor.java:1033)
at org.jboss.cache.interceptors.OrderedSynchronizationHandler.beforeCompletion(OrderedSynchronizationHandler.java:72)
at org.jboss.tm.TransactionImpl.doBeforeCompletion(TransactionImpl.java:1473)
at org.jboss.tm.TransactionImpl.beforePrepare(TransactionImpl.java:1092)
at org.jboss.tm.TransactionImpl.commit(TransactionImpl.java:306)
... 86 more
Caused by: org.jboss.cache.ReplicationException: rsp=sender=172.16.13.67:32865, retval=null, received=false, suspected=false
at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:3747)
at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:3672)
at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:3770)
at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:87)
at org.jboss.cache.interceptors.ReplicationInterceptor.runPreparePhase(ReplicationInterceptor.java:133)
at org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:52)
at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:67)
at org.jboss.cache.interceptors.TxInterceptor.runPreparePhase(TxInterceptor.java:753)
at org.jboss.cache.interceptors.TxInterceptor$LocalSynchronizationHandler.beforeCompletion(TxInterceptor.java:1011)
... 90 more
Caused by: org.jboss.cache.lock.TimeoutException: timeout for 172.16.13.67:32865
at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:3745)

This is how our JBoss Cache configuration looks like.

<mbean code="org.jboss.cache.TreeCache"
 name="jboss.cache:service=ClusteredTreeCache">

 <depends>jboss:service=Naming</depends>
 <depends>jboss:service=TransactionManager</depends>

 <attribute name="TransactionManagerLookupClass">org.jboss.cache.JBossTransactionManagerLookup</attribute>

 <attribute name="IsolationLevel">REPEATABLE_READ</attribute>

 <attribute name="CacheMode">REPL_SYNC</attribute>

 <attribute name="UseMarshalling">true</attribute>
 <attribute name="InactiveOnStartup">true</attribute>

 <attribute name="ClusterName">Cache-Cluster</attribute>


 <attribute name="ClusterConfig">
 <config>
 <UDP mcast_addr="228.1.2.18" mcast_port="45579"
 ip_ttl="8" ip_mcast="true"
 mcast_send_buf_size="150000" mcast_recv_buf_size="80000"
 ucast_send_buf_size="150000" ucast_recv_buf_size="80000"
 loopback="false"/>
 <PING timeout="2000" num_initial_members="3"
 up_thread="false" down_thread="false"/>
 <MERGE2 min_interval="10000" max_interval="20000"/>
 <FD_SOCK/>
 <VERIFY_SUSPECT timeout="1500"
 up_thread="false" down_thread="false"/>
 <pbcast.NAKACK gc_lag="50" retransmit_timeout="600,1200,2400,4800"
 max_xmit_size="8192" up_thread="false" down_thread="false"/>
 <UNICAST timeout="600,1200,2400" window_size="100" min_threshold="10"
 down_thread="false"/>
 <pbcast.STABLE desired_avg_gossip="20000"
 up_thread="false" down_thread="false"/>
 <FRAG frag_size="8192"
 down_thread="false" up_thread="false"/>
 <pbcast.GMS join_timeout="5000" join_retry_timeout="2000"
 shun="true" print_local_addr="true"/>
 <pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/>
 </config>

 </attribute>

 <attribute name="LockAcquisitionTimeout">15000</attribute>

 </mbean>

 </server>

[ben.wang]

If you are running under high load, is REPL_ASYNC an option? It is much performant. But you will also need (flow control) in the stack. See here for details:

http://wiki.jboss.org/wiki/Wiki.jsp?page=HighPerformanceSessionReplicationConfigAsyncMode

[manik]

Are you using 1.3.0.SP2?

[manishobhatia]

No Manik I am using 1.3.0. will using SP2 help ?

Also we are using cache in our production website, and it is important to have the cache sync'd up at all times. Thats why we are using REPL_SYNC.

Ours is a CMS system where all the website pages are retrieved from the database and cached using JBoss Cache.
We face this problem mostly at startup when the cache is being built. That means there are many simultaneous Put / Get calls on the cache.

Once the cache is built, this problem is reduced. As there are only Get calls on cache

Is there anything else I can change in my configuration to get better performance and reduce locking errors in such conditions.

Thanks,
Manish

[manik]

Yes - SP2 did fix some issues around transaction handling and this may apply to your case.