"Response timed out" exception in TreeCache
Hi, folks.
On our environment occasionally were observed numerous org.jboss.cache.lock.TimeoutException's that cause transactions to fail. Moreover the problem seems to progress over time resulting in a system becoming non-operational with restart required.
It was suggested to try to increase SynReplTimeOut up to 100 sec and set loopback attribute to true (something that is recommended on windows machines). As issue reproduces occasionally it is hard to verify that these changes will help.
Any thoughts regarding this problem are appreciated.
We use:
JBossAS - 4.2.2.GA
The JBossCache - 1.4.1.SP5
JGroups - 2.4.1-SP4
The config
<mbean code="org.jboss.cache.TreeCache" name="jboss.cache:service=HibernateTreeCache"> <depends>jboss:service=Naming</depends> <depends>jboss:service=TransactionManager</depends> <attribute name="TransactionManagerLookupClass">org.jboss.cache.JBossTransactionManagerLookup</attribute> <attribute name="IsolationLevel">REPEATABLE_READ</attribute> <attribute name="CacheMode">REPL_SYNC</attribute> <attribute name="ClusterName">DEFAULT</attribute> <attribute name="ClusterConfig"> <config> <UDP mcast_addr="231.12.21.132" mcast_port="45047" ip_ttl="64" ip_mcast="true" mcast_send_buf_size="150000" mcast_recv_buf_size="80000" ucast_send_buf_size="150000" ucast_recv_buf_size="80000" loopback="false"/> <PING timeout="2000" num_initial_members="3" up_thread="false" down_thread="false"/> <MERGE2 min_interval="10000" max_interval="20000"/> <FD shun="true" up_thread="true" down_thread="true"/> <VERIFY_SUSPECT timeout="1500" up_thread="false" down_thread="false"/> <pbcast.NAKACK gc_lag="50" retransmit_timeout="600,1200,2400,4800" up_thread="false" down_thread="false"/> <pbcast.STABLE desired_avg_gossip="20000" up_thread="false" down_thread="false"/> <UNICAST timeout="600,1200,2400" window_size="100" min_threshold="10" down_thread="false"/> <FRAG frag_size="8192" down_thread="false" up_thread="false"/> <pbcast.GMS join_timeout="5000" join_retry_timeout="2000" shun="true" print_local_addr="true"/> <pbcast.STATE_TRANSFER up_thread="false" down_thread="false"/> </config> </attribute> <attribute name="InitialStateRetrievalTimeout">20000</attribute> <attribute name="SyncReplTimeout">10000</attribute> <attribute name="LockAcquisitionTimeout">15000</attribute> <attribute name="EvictionPolicyClass"></attribute> </mbean>
the stack trace:
2008-09-15 04:44:32,548 WARN [http-0.0.0.0-80-2] [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator_2] TwoPhaseCoordinator.beforeCompletion - failed for com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple@f09a652 java.lang.RuntimeException: at org.jboss.cache.interceptors.TxInterceptor$LocalSynchronizationHandler.beforeCompletion(TxInterceptor.java:1182) at org.jboss.cache.interceptors.OrderedSynchronizationHandler.beforeCompletion(OrderedSynchronizationHandler.java:75) at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:114) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:247) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:86) at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:177) at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1389) at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:135) at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:87) at org.jboss.tm.usertx.client.ServerVMClientUserTransaction.commit(ServerVMClientUserTransaction.java:140) at com.enkata.common.transaction.TransactionManager.commit(TransactionManager.java:75) at com.enkata.common.transaction.TransactionManagerHelper.commit(TransactionManagerHelper.java:43) at com.enkata.common.transaction.ATransaction.run(ATransaction.java:91) at com.enkata.common.transaction.NonCancellableTransaction.run(NonCancellableTransaction.java:33) at com.enkata.common.transaction.ReturnObjectTransaction.executeTransaction(ReturnObjectTransaction.java:43) at com.enkata.eac.to.analysis.AnalysisUIFacade.drill(AnalysisUIFacade.java:171) at com.enkata.eac.ui.analyst.drillpath.DrillPathNavigationCmd.doRun(DrillPathNavigationCmd.java:72) at com.enkata.eac.ui.util.ACmd.run(ACmd.java:70) at com.enkata.eac.ui.framework.Page.runACmd(Page.java:746) at com.enkata.eac.ui.framework.Page.runCmdAndAdjustScrollTop(Page.java:209) at com.enkata.eac.ui.framework.Page.validateAndUpdate(Page.java:192) at com.enkata.eac.ui.framework.Controller.validateAndUpdate(Controller.java:373) at com.enkata.eac.ui.framework.Controller.process2(Controller.java:270) at com.enkata.eac.ui.framework.Controller.process(Controller.java:250) at com.enkata.eac.ui.framework.UIMgr.processRequest(UIMgr.java:133) at com.enkata.eac.ui.servlet.EACServlet.processRequest(EACServlet.java:91) at com.enkata.eac.ui.servlet.EACServlet.doPost(EACServlet.java:67) at javax.servlet.http.HttpServlet.service(HttpServlet.java:710) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.enkata.eac.ui.servlet.DownloadExportFilter.doFilter(DownloadExportFilter.java:41) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83) at org.jboss.seam.web.MultipartFilter.doFilter(MultipartFilter.java:85) at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) at org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64) at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) at org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45) at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) at org.ajax4jsf.webapp.BaseXMLFilter.doXmlFilter(BaseXMLFilter.java:147) at org.ajax4jsf.webapp.BaseFilter.doFilter(BaseFilter.java:276) at org.ajax4jsf.Filter.doFilter(Filter.java:175) at org.jboss.seam.web.Ajax4jsfFilter.doFilter(Ajax4jsfFilter.java:60) at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) at org.jboss.seam.web.LoggingFilter.doFilter(LoggingFilter.java:58) at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) at org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446) at java.lang.Thread.run(Unknown Source) Caused by: org.jboss.cache.ReplicationException: rsp=sender=172.16.100.17:3216, retval=null, received=false, suspected=false at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4436) at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4358) at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4469) at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:110) at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:88) at org.jboss.cache.interceptors.ReplicationInterceptor.runPreparePhase(ReplicationInterceptor.java:152) at org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:69) at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68) at org.jboss.cache.interceptors.TxInterceptor.runPreparePhase(TxInterceptor.java:873) at org.jboss.cache.interceptors.TxInterceptor$LocalSynchronizationHandler.beforeCompletion(TxInterceptor.java:1156) ... 66 more Caused by: org.jboss.cache.lock.TimeoutException: Response timed out: sender=172.16.100.17:3216, retval=null, received=false, suspected=false at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4434) ... 75 more
