3 Replies Latest reply on May 4, 2015 7:45 AM by rvansa

    Timed out waiting for the state transfer lock, state transfer in progress for view 11

    gspadmapriya
    gspadmapriya Apr 27, 2015 4:24 PM

      Timed out waiting for the state transfer lock, state transfer in progress for view 11

          at com.authentication.provider.OAuthProviderExceptionHandler.convertToProviderException(OAuthProviderExceptionHandler.java:59)

          at com.authentication.provider.OAuthProviderExceptionHandler.handle(OAuthProviderExceptionHandler.java:68)

          at com.authentication.provider.impl.BaseOAuthProvider.generateAndCacheRequestToken(BaseOAuthProvider.java:128)

          at com.authentication.service.impl.AuthenticationServiceImpl.generateAndCacheRequestToken(AuthenticationServiceImpl.java:74)

          at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)

          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

          at java.lang.reflect.Method.invoke(Method.java:597)

          at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)

          at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)

          at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)

          at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)

          at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

          at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)

          at $Proxy12.generateAndCacheRequestToken(Unknown Source)

          at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)

          at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)

          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

          at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)

          at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

          at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

          at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)

          at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

          at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)

          at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)

          at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)

          at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)

          at java.lang.Thread.run(Thread.java:662)

      Caused by: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 11

          at org.infinispan.interceptors.StateTransferLockInterceptor.signalStateTransferInProgress(StateTransferLockInterceptor.java:200)

          at org.infinispan.interceptors.StateTransferLockInterceptor.handleWriteCommand(StateTransferLockInterceptor.java:176)

          at org.infinispan.interceptors.StateTransferLockInterceptor.visitPutKeyValueCommand(StateTransferLockInterceptor.java:151)

          at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:133)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)

          at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)

          at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)

          at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345)

          at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1007)

          at org.infinispan.CacheImpl.put(CacheImpl.java:704)

          at org.infinispan.CacheImpl.put(CacheImpl.java:696)

          at org.infinispan.CacheSupport.put(CacheSupport.java:53)

          at com.directv.authentication.service.impl.ProviderTokenServiceImpl.cacheRequestToken(ProviderTokenServiceImpl.java:48)

          at sun.reflect.GeneratedMethodAccessor53.invoke(Unknown Source)

          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

          at java.lang.reflect.Method.invoke(Method.java:597)

          at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)

          at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)

          at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)

          at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)

          at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

          at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)

          at $Proxy30.cacheRequestToken(Unknown Source)

       

       

      Can anyone help me ?  I have attached jgroups config file. My infinispan config file is like

        <namedCache name="requestTokenCache">

          <clustering mode="dist">

            <l1 enabled="true" lifespan="60000"/>       

            <stateTransfer fetchInMemoryState="true" timeout="30000"/>      

            <sync/>

          </clustering>         

          <expiration lifespan="60000" wakeUpInterval="30000"/>    

          <eviction maxEntries="1000" strategy="LRU"/>       

        </namedCache>    

      • 900 Views
      • Tags:
        • 1. Re: Timed out waiting for the state transfer lock, state transfer in progress for view 11
          rvansa
          rvansa Apr 29, 2015 2:41 AM (in response to gspadmapriya)

          So the question is why the state transfer is not finished in time. It either may be a bug (state transfer is stuck), or it just takes long - btw., is this version 5.1.x? In that version, no writes could be executed until the data are moved between nodes. So, just wait. You could increase the timeout, but robust code should always catch exceptions from cache calls (and either let user start the request again, or wait a random period and retry).

          However, don't expect much help with 3 years old version. Upgrade and then we can try to fix latest version.

          1 of 1 people found this helpful
            • Actions
          • 2. Re: Timed out waiting for the state transfer lock, state transfer in progress for view 11
            gspadmapriya
            gspadmapriya Apr 30, 2015 11:40 AM (in response to rvansa)

            I have upgraded infinispan to 7.0.3.Appended configuration file at the end, both infinispan.xml and jgroups.xml

            I am getting the following 2 exceptions - replication exception and Timeout exception, Can you please help me, to set the right config. Please let me know, the exactly, which attribute to set to what value. Thanks!

            ----------------------Exception 1-----------replication exception:

            [04/30/15 03:38:17PM] [WARN ] [Transaction Reaper] arjuna - ARJUNA012117: TransactionReaper::check timeout for TX 0:ffffac1f1fcd:e18c:55411ff1:1409 in state  RUN

            [04/30/15 03:38:17PM] [TRACE] [Transaction Reaper] arjuna - Reaper scheduling TX for cancellation 0:ffffac1f1fcd:e18c:55411ff1:1409

            [04/30/15 03:38:17PM] [TRACE] [Transaction Reaper] arjuna - TransactionReaper::check - comparing 1430408298215

            [04/30/15 03:38:17PM] [TRACE] [http-8080-117] RpcManagerImpl - replication exception:

            org.infinispan.remoting.transport.jgroups.SuspectException: One or more nodes have left the cluster while replicating command SingleRpcCommand{cacheName='requestTokenCache', command=PutKeyValueCommand{key=facebook.stb.123.123, value=OAuthProviderToken[date=Thu Apr 30 15:38:17 UTC 2015,decodedToken=facebook,1430408297714,encodedToken=WyJmYWNlYm9vayIsIjE0MzA0MDgyOTc3MTQiXQ,providerId=facebook,sessionId=<null>], flags=null, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedMetadata{version=null}, successful=true}}

                at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:523)

                at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:290)

                at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:272)

                at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:93)

                at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

                at org.infinispan.interceptors.distribution.L1NonTxInterceptor.handleDataWriteCommand(L1NonTxInterceptor.java:297)

                at org.infinispan.interceptors.distribution.L1NonTxInterceptor.visitPutKeyValueCommand(L1NonTxInterceptor.java:208)

                at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

                at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:386)

                at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:474)

                at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:187)

                at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

                at org.infinispan.interceptors.distribution.L1LastChanceInterceptor.visitDataWriteCommand(L1LastChanceInterceptor.java:79)

                at org.infinispan.interceptors.distribution.L1LastChanceInterceptor.visitPutKeyValueCommand(L1LastChanceInterceptor.java:65)

                at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

                at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:46)

                at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

                at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)

                at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:35)

                at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

                at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)

                at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:35)

                at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

                at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:187)

                at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:108)

                at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

                at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:159)

                at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:145)

                at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

                at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)

                at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)

                at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:35)

                at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)

                at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)

                at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1578)

                at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:1056)

                at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1048)

                at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1649)

                at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:246)

                at com.directv.authentication.service.impl.ProviderTokenServiceImpl.cacheRequestToken(ProviderTokenServiceImpl.java:41)

                at com.directv.authentication.provider.impl.BaseOAuthProvider.cacheRequestToken(BaseOAuthProvider.java:318)

                at com.directv.authentication.provider.impl.BaseOAuthProvider.generateAndCacheRequestToken(BaseOAuthProvider.java:116)

                at com.directv.authentication.service.impl.AuthenticationServiceImpl.generateAndCacheRequestToken(AuthenticationServiceImpl.java:74)

                at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                at java.lang.reflect.Method.invoke(Method.java:606)

                at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)

                at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)

                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)

                at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)

                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

                at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)

                at com.sun.proxy.$Proxy12.generateAndCacheRequestToken(Unknown Source)

                at com.directv.socialnetwork.web.RequestTokenServlet.doPost(RequestTokenServlet.java:109)

                at com.directv.socialnetwork.web.RequestTokenServlet.doGet(RequestTokenServlet.java:58)

                at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)

                at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)

                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

                at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)

                at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

                at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

                at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)

                at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:615)

                at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

                at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)

                at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)

                at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)

                at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)

                at java.lang.Thread.run(Thread.java:745)

             

             

            -------------------------------------------------------------Exception 1----------org.infinispan.util.concurrent.TimeoutException

            [04/30/15 03:33:07PM] [ERROR] [http-8080-111] InvocationContextInterceptor - ISPN000136: Execution error

            org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 115

                at org.infinispan.statetransfer.StateTransferLockImpl.waitForTransactionData(StateTransferLockImpl.java:93)

                at org.infinispan.interceptors.base.BaseStateTransferInterceptor.waitForTransactionData(BaseStateTransferInterceptor.java:96)

                at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:202)

                at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:108)

                at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

                at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:159)

                at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:145)

                at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

                at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)

                at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)

                at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:35)

                at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)

                at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)

                at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1578)

                at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:1056)

                at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1048)

                at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1649)

                at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:246)

                at com.directv.authentication.service.impl.ProviderTokenServiceImpl.cacheRequestToken(ProviderTokenServiceImpl.java:41)

                at com.directv.authentication.provider.impl.BaseOAuthProvider.cacheRequestToken(BaseOAuthProvider.java:318)

                at com.directv.authentication.provider.impl.BaseOAuthProvider.generateAndCacheRequestToken(BaseOAuthProvider.java:116)

                at com.directv.authentication.service.impl.AuthenticationServiceImpl.generateAndCacheRequestToken(AuthenticationServiceImpl.java:74)

                at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                at java.lang.reflect.Method.invoke(Method.java:606)

                at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)

                at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)

                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)

                at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)

                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

                at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)

                at com.sun.proxy.$Proxy12.generateAndCacheRequestToken(Unknown Source)

                at com.directv.socialnetwork.web.RequestTokenServlet.doPost(RequestTokenServlet.java:109)

                at com.directv.socialnetwork.web.RequestTokenServlet.doGet(RequestTokenServlet.java:58)

                at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)

                at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)

                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

                at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)

                at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

                at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

                at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)

                at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:615)

                at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

                at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)

                at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)

                at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)

                at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)

                at java.lang.Thread.run(Thread.java:745)

            [04/30/15 03:33:07PM] [TRACE] [http-8080-111] jta - BaseTransaction.getStatus

            [04/30/15 03:33:07PM] [TRACE] [http-8080-111] jta - TransactionImple.getStatus

            [04/30/15 03:33:07PM] [DEBUG] [http-8080-111] JtaTransactionManager - Initiating transaction commit

            [04/30/15 03:33:07PM] [TRACE] [http-8080-111] jta - BaseTransaction.getStatus

            [04/30/15 03:33:07PM] [TRACE] [http-8080-111] jta - TransactionImple.getStatus

            [04/30/15 03:33:07PM] [TRACE] [http-8080-111] jta - BaseTransaction.rollback

            [04/30/15 03:33:07PM] [TRACE] [http-8080-111] jta - TransactionImple.rollbackAndDisassociate

            [04/30/15 03:33:07PM] [TRACE] [http-8080-111] arjuna - BasicAction::Abort() for action-id 0:ffffac1f1fcd:e18c:55411ff1:13eb

            [04/30/15 03:33:07PM] [WARN ] [http-8080-111] arjuna - ARJUNA012077: Abort called on already aborted atomic action 0:ffffac1f1fcd:e18c:55411ff1:13eb

            [04/30/15 03:33:07PM] [TRACE] [http-8080-111] arjuna - BasicAction::removeChildThread () action 0:ffffac1f1fcd:e18c:55411ff1:13eb removing TSThread:70

            [04/30/15 03:33:07PM] [TRACE] [http-8080-111] arjuna - BasicAction::removeChildThread () action 0:ffffac1f1fcd:e18c:55411ff1:13eb removing TSThread:70 result = true

            [04/30/15 03:33:07PM] [TRACE] [http-8080-111] arjuna - TransactionReaper::remove ( BasicAction: 0:ffffac1f1fcd:e18c:55411ff1:13eb status: ActionStatus.ABORTED )

            [04/30/15 03:33:07PM] [ERROR] [http-8080-111] TransactionInterceptor - Application exception overridden by commit exception

            com.directv.authentication.provider.OAuthProviderException: Timed out waiting for topology 115

                at com.directv.authentication.provider.OAuthProviderExceptionHandler.convertToProviderException(OAuthProviderExceptionHandler.java:59)

                at com.directv.authentication.provider.OAuthProviderExceptionHandler.handle(OAuthProviderExceptionHandler.java:68)

                at com.directv.authentication.provider.impl.BaseOAuthProvider.generateAndCacheRequestToken(BaseOAuthProvider.java:128)

                at com.directv.authentication.service.impl.AuthenticationServiceImpl.generateAndCacheRequestToken(AuthenticationServiceImpl.java:74)

                at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                at java.lang.reflect.Method.invoke(Method.java:606)

                at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)

                at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)

                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)

                at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)

                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

                at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)

                at com.sun.proxy.$Proxy12.generateAndCacheRequestToken(Unknown Source)

                at com.directv.socialnetwork.web.RequestTokenServlet.doPost(RequestTokenServlet.java:109)

                at com.directv.socialnetwork.web.RequestTokenServlet.doGet(RequestTokenServlet.java:58)

                at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)

                at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)

                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

                at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)

                at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

                at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

                at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)

                at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:615)

                at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

                at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)

                at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)

                at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)

                at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)

                at java.lang.Thread.run(Thread.java:745)

            Caused by: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 115

                at org.infinispan.statetransfer.StateTransferLockImpl.waitForTransactionData(StateTransferLockImpl.java:93)

                at org.infinispan.interceptors.base.BaseStateTransferInterceptor.waitForTransactionData(BaseStateTransferInterceptor.java:96)

                at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:202)

                at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:108)

                at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

                at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:159)

                at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:145)

                at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

                at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)

                at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)

                at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:35)

                at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)

                at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)

                at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1578)

                at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:1056)

                at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1048)

                at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1649)

                at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:246)

                at com.directv.authentication.service.impl.ProviderTokenServiceImpl.cacheRequestToken(ProviderTokenServiceImpl.java:41)

                at com.directv.authentication.provider.impl.BaseOAuthProvider.cacheRequestToken(BaseOAuthProvider.java:318)

                at com.directv.authentication.provider.impl.BaseOAuthProvider.generateAndCacheRequestToken(BaseOAuthProvider.java:116)

                ... 28 more

            [04/30/15 03:33:07PM] [ERROR] [http-8080-111] RequestTokenServlet - JTA transaction already rolled back (probably due to a timeout)

            org.springframework.transaction.UnexpectedRollbackException: JTA transaction already rolled back (probably due to a timeout)

                at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1008)

                at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:754)

                at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)

                at org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:432)

                at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:114)

                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

                at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)

                at com.sun.proxy.$Proxy12.generateAndCacheRequestToken(Unknown Source)

                at com.directv.socialnetwork.web.RequestTokenServlet.doPost(RequestTokenServlet.java:109)

                at com.directv.socialnetwork.web.RequestTokenServlet.doGet(RequestTokenServlet.java:58)

                at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)

                at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)

                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

                at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)

                at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

                at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

                at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)

                at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:615)

                at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

                at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)

                at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)

                at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)

                at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)

                at java.lang.Thread.run(Thread.java:745)

             

            --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

            Config files

            infinispan.xml--------------

            <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

                        xmlns="urn:infinispan:config:7.0"

                        xsi:schemaLocation="

                          urn:infinispan:config:7.0 http://www.infinispan.org/schemas/infinispan-config-7.0.xsd

                        "

            >

            <jgroups>

            <stack-file name="tcp" path="com.directv.socialnetwork/jgroups_static.xml" />

            </jgroups>

             

            <cache-container default-cache="default">

            <transport cluster="socialnetwork1" stack="tcp"></transport>

            <distributed-cache name="requestTokenCache" mode="SYNC" l1-lifespan="300000" statistics="true" owners="2" remote-timeout="200000" />

            <distributed-cache name="accessTokenCache" mode="SYNC" l1-lifespan="300000" statistics="true" owners="2" remote-timeout="200000"/>

            <distributed-cache name="globalFeedCache" mode="SYNC" l1-lifespan="300000" statistics="true" owners="2" remote-timeout="200000"/>

            <distributed-cache name="linkCache" mode="SYNC" l1-lifespan="300000" statistics="true" owners="2" remote-timeout="200000"/>

            <distributed-cache name="imageUrlCollectionCache" mode="SYNC" l1-lifespan="300000" statistics="true" owners="2" remote-timeout="200000"/>

             

            <local-cache name="programCache" statistics="true" />

            <local-cache name="programCollectionCache" statistics="true"/>

            </cache-container>

             

             

            </infinispan>

             

            -----------------------jgroups_static.xml, using TCP

            <config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

                    xmlns="urn:org:jgroups"

                    xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">

                <TCP bind_addr="SITE_LOCAL"

                bind_port="${jgroups.tcp.port:7800}"

                loopback="true"

                port_range="2"

                     recv_buf_size="${tcp.recv_buf_size:5M}"

                     send_buf_size="${tcp.send_buf_size:5M}"

                     max_bundle_size="64K"

                     max_bundle_timeout="30"

                     use_send_queues="true"

                     sock_conn_timeout="300"

             

                     timer_type="new3"

                     timer.min_threads="4"

                     timer.max_threads="10"

                     timer.keep_alive_time="3000"

                     timer.queue_max_size="500"

                     thread_pool.enabled="true"

                     thread_pool.min_threads="2"

                     thread_pool.max_threads="8"

                     thread_pool.keep_alive_time="5000"

                     thread_pool.queue_enabled="true"

                     thread_pool.queue_max_size="10000"

                     thread_pool.rejection_policy="discard"

             

                     oob_thread_pool.enabled="true"

                     oob_thread_pool.min_threads="2"

                     oob_thread_pool.max_threads="8"

                     oob_thread_pool.keep_alive_time="5000"

                     oob_thread_pool.queue_enabled="false"

                     oob_thread_pool.queue_max_size="100"

                     oob_thread_pool.rejection_policy="discard"/>

                                     

                <TCPPING async_discovery="true"

                         initial_hosts="172.31.31.205[7800],172.31.22.195[7800],172.31.18.195[7800],172.31.18.169[7800],172.31.18.178[7800],172.31.18.192[7800],172.31.18.183[7800],172.31.18.191[7800],172.31.18.190[7800],172.31.18.199[7800],172.31.18.174[7800],172.31.18.179[7800],172.31.18.184[7800],172.31.18.186[7800],172.31.18.198[7800],172.31.18.201[7800],172.31.18.196[7800],172.31.18.170[7800],172.31.18.172[7800],172.31.18.181[7800],172.31.18.173[7800],172.31.18.188[7800],172.31.18.187[7800],172.31.18.182[7800],172.31.18.189[7800],172.31.18.177[7800],172.31.18.180[7800],172.31.18.171[7800],172.31.18.185[7800],172.31.18.200[7800],172.31.18.176[7800],172.31.18.194[7800],172.31.18.168[7800],172.31.18.193[7800],172.31.18.175[7800],172.31.18.197[7800]"

                         port_range="2"/>

                <MERGE3  min_interval="10000"

                         max_interval="30000"/>

                <FD_SOCK/>

                <FD timeout="3000" max_tries="3" />

                <VERIFY_SUSPECT timeout="1500"  />

                <BARRIER />

                <pbcast.NAKACK2 use_mcast_xmit="false"

                               discard_delivered_msgs="true"/>

                <UNICAST3 />

                <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"

                               max_bytes="4M"/>

                <pbcast.GMS print_local_addr="true" join_timeout="2000"

                            view_bundling="true"/>

                <MFC max_credits="2M"

                     min_threshold="0.4"/>

                <FRAG2 frag_size="60K"  />

                <!--RSVP resend_interval="2000" timeout="10000"/-->

                <pbcast.STATE_TRANSFER/>

            </config>

              • Actions
            • 3. Re: Timed out waiting for the state transfer lock, state transfer in progress for view 11
              rvansa
              rvansa May 4, 2015 7:45 AM (in response to gspadmapriya)

              So the first exception, SuspectException is a hint what happened. One of the nodes got suspected (other nodes thought that it crashed), probably due to GC pause. You need to find out your maximum GC pause duration (I think you've posted the times in another thread) and set the timeout for suspection to value larger than that. Your JGroups config shows that you have FD.timeout set to 3 seconds, which is probably too low value. It's good that you use VERIFY_SUSPECT, but out of my experience I would increase the timeout to 5 seconds, too.

               

              The drawback of long timeouts is that if a node really crashes, you don't detect that for quite a long time, and all write requests for entries hosted on the crashed machine are stuck.

               

              Btw., I usually use FD_ALL instead of FD to detect crash sooner in case of multiple crashed nodes - FD uses ring topology of checking liveness (see JGroups docs for details), therefore if N nodes crash, you may detect that only after N * timeout.

                • Actions