2 Replies Latest reply on Oct 1, 2008 2:07 AM by Denis S. Kuchugurov

    "Response timed out" exception in TreeCache

    Denis S. Kuchugurov Newbie

      Hi, folks.

      On our environment occasionally were observed numerous org.jboss.cache.lock.TimeoutException's that cause transactions to fail. Moreover the problem seems to progress over time resulting in a system becoming non-operational with restart required.

      It was suggested to try to increase SynReplTimeOut up to 100 sec and set loopback attribute to true (something that is recommended on windows machines). As issue reproduces occasionally it is hard to verify that these changes will help.

      Any thoughts regarding this problem are appreciated.

      We use:
      JBossAS - 4.2.2.GA
      The JBossCache - 1.4.1.SP5
      JGroups - 2.4.1-SP4

      The config

      <mbean code="org.jboss.cache.TreeCache" name="jboss.cache:service=HibernateTreeCache">
       <depends>jboss:service=Naming</depends>
       <depends>jboss:service=TransactionManager</depends>
       <attribute name="TransactionManagerLookupClass">org.jboss.cache.JBossTransactionManagerLookup</attribute>
       <attribute name="IsolationLevel">REPEATABLE_READ</attribute>
       <attribute name="CacheMode">REPL_SYNC</attribute>
       <attribute name="ClusterName">DEFAULT</attribute>
      
       <attribute name="ClusterConfig">
       <config>
       <UDP mcast_addr="231.12.21.132" mcast_port="45047"
       ip_ttl="64" ip_mcast="true"
       mcast_send_buf_size="150000" mcast_recv_buf_size="80000"
       ucast_send_buf_size="150000" ucast_recv_buf_size="80000"
       loopback="false"/>
       <PING timeout="2000" num_initial_members="3"
       up_thread="false" down_thread="false"/>
       <MERGE2 min_interval="10000" max_interval="20000"/>
       <FD shun="true" up_thread="true" down_thread="true"/>
       <VERIFY_SUSPECT timeout="1500"
       up_thread="false" down_thread="false"/>
       <pbcast.NAKACK gc_lag="50" retransmit_timeout="600,1200,2400,4800"
       up_thread="false" down_thread="false"/>
       <pbcast.STABLE desired_avg_gossip="20000"
       up_thread="false" down_thread="false"/>
       <UNICAST timeout="600,1200,2400" window_size="100" min_threshold="10"
       down_thread="false"/>
       <FRAG frag_size="8192"
       down_thread="false" up_thread="false"/>
       <pbcast.GMS join_timeout="5000" join_retry_timeout="2000"
       shun="true" print_local_addr="true"/>
       <pbcast.STATE_TRANSFER up_thread="false" down_thread="false"/>
       </config>
       </attribute>
       <attribute name="InitialStateRetrievalTimeout">20000</attribute>
       <attribute name="SyncReplTimeout">10000</attribute>
       <attribute name="LockAcquisitionTimeout">15000</attribute>
       <attribute name="EvictionPolicyClass"></attribute>
       </mbean>


      the stack trace:
      2008-09-15 04:44:32,548 WARN [http-0.0.0.0-80-2] [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator_2] TwoPhaseCoordinator.beforeCompletion - failed for com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple@f09a652
      java.lang.RuntimeException:
       at org.jboss.cache.interceptors.TxInterceptor$LocalSynchronizationHandler.beforeCompletion(TxInterceptor.java:1182)
       at org.jboss.cache.interceptors.OrderedSynchronizationHandler.beforeCompletion(OrderedSynchronizationHandler.java:75)
       at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:114)
       at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:247)
       at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:86)
       at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:177)
       at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1389)
       at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:135)
       at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:87)
       at org.jboss.tm.usertx.client.ServerVMClientUserTransaction.commit(ServerVMClientUserTransaction.java:140)
       at com.enkata.common.transaction.TransactionManager.commit(TransactionManager.java:75)
       at com.enkata.common.transaction.TransactionManagerHelper.commit(TransactionManagerHelper.java:43)
       at com.enkata.common.transaction.ATransaction.run(ATransaction.java:91)
       at com.enkata.common.transaction.NonCancellableTransaction.run(NonCancellableTransaction.java:33)
       at com.enkata.common.transaction.ReturnObjectTransaction.executeTransaction(ReturnObjectTransaction.java:43)
       at com.enkata.eac.to.analysis.AnalysisUIFacade.drill(AnalysisUIFacade.java:171)
       at com.enkata.eac.ui.analyst.drillpath.DrillPathNavigationCmd.doRun(DrillPathNavigationCmd.java:72)
       at com.enkata.eac.ui.util.ACmd.run(ACmd.java:70)
       at com.enkata.eac.ui.framework.Page.runACmd(Page.java:746)
       at com.enkata.eac.ui.framework.Page.runCmdAndAdjustScrollTop(Page.java:209)
       at com.enkata.eac.ui.framework.Page.validateAndUpdate(Page.java:192)
       at com.enkata.eac.ui.framework.Controller.validateAndUpdate(Controller.java:373)
       at com.enkata.eac.ui.framework.Controller.process2(Controller.java:270)
       at com.enkata.eac.ui.framework.Controller.process(Controller.java:250)
       at com.enkata.eac.ui.framework.UIMgr.processRequest(UIMgr.java:133)
       at com.enkata.eac.ui.servlet.EACServlet.processRequest(EACServlet.java:91)
       at com.enkata.eac.ui.servlet.EACServlet.doPost(EACServlet.java:67)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
       at com.enkata.eac.ui.servlet.DownloadExportFilter.doFilter(DownloadExportFilter.java:41)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
       at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83)
       at org.jboss.seam.web.MultipartFilter.doFilter(MultipartFilter.java:85)
       at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
       at org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64)
       at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
       at org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45)
       at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
       at org.ajax4jsf.webapp.BaseXMLFilter.doXmlFilter(BaseXMLFilter.java:147)
       at org.ajax4jsf.webapp.BaseFilter.doFilter(BaseFilter.java:276)
       at org.ajax4jsf.Filter.doFilter(Filter.java:175)
       at org.jboss.seam.web.Ajax4jsfFilter.doFilter(Ajax4jsfFilter.java:60)
       at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
       at org.jboss.seam.web.LoggingFilter.doFilter(LoggingFilter.java:58)
       at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
       at org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
       at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
       at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
       at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
       at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
       at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
       at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
       at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
       at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
       at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
       at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
       at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
       at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
       at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
       at java.lang.Thread.run(Unknown Source)
      Caused by: org.jboss.cache.ReplicationException: rsp=sender=172.16.100.17:3216, retval=null, received=false, suspected=false
       at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4436)
       at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4358)
       at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4469)
       at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:110)
       at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:88)
       at org.jboss.cache.interceptors.ReplicationInterceptor.runPreparePhase(ReplicationInterceptor.java:152)
       at org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:69)
       at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
       at org.jboss.cache.interceptors.TxInterceptor.runPreparePhase(TxInterceptor.java:873)
       at org.jboss.cache.interceptors.TxInterceptor$LocalSynchronizationHandler.beforeCompletion(TxInterceptor.java:1156)
       ... 66 more
      Caused by: org.jboss.cache.lock.TimeoutException: Response timed out: sender=172.16.100.17:3216, retval=null, received=false, suspected=false
       at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4434)
       ... 75 more