7 Replies Latest reply on May 10, 2017 5:26 PM by Purushothaman S

    Issues with Infinispan Partition Handling

    Purushothaman S Newbie

      Hi,

       

      We have an Infinispan server cluster (cluster size 3). We have defined a cache container and a replicated cache with partition handling enabled in the default cluster configuration file (clustered.xml). We observed that when shut down 2 of the Infinispan servers and bring them back up, they fail to join the cluster due to State Transfer timeout  error. The state transfer timeout which we have configured for the cache is 5 mins. The error is observed even when the cache is empty hence no reason to increase the timeout.

       

      We also verified that the issue occurs only when partition handling is enabled. The Infinispan server version which we have deployed is 8.2.6.

       

      We use TCPPING as the discovery protocol to form the cluster.

       

      Cache configuration:

       

          <transport lock-timeout="60000"/>

          <replicated-cache name="default" mode="SYNC" batching="true">

              <partition-handling enabled="true"/>

              <locking isolation="REPEATABLE_READ"/>

              <state-transfer timeout="300000"/>

          </replicated-cache>

       

      The following exception is observed in the nodes which fail to rejoin the cluster:

       

      27-Apr-2017 22:21:34,985 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-7) <:> MSC000001: Failed to start service jboss.datagrid-infinispan.vsd.default: org.jboss.msc.service.StartException in service jboss.datagrid-infinispan.vsd.default: Failed to start service

              at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1904) [jboss-msc-1.2.6.Final.jar:1.2.6.Final]

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_111]

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_111]

              at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_111]

      Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl

              at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:172)

              at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:859)

              at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:628)

              at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:617)

              at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:542)

              at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:238)

              at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:862)

              at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:635)

              at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:585)

              at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:451)

              at org.infinispan.manager.impl.AbstractDelegatingEmbeddedCacheManager.getCache(AbstractDelegatingEmbeddedCacheManager.java:133)

              at org.infinispan.server.infinispan.SecurityActions$5.run(SecurityActions.java:130)

              at org.infinispan.server.infinispan.SecurityActions$5.run(SecurityActions.java:127)

              at org.infinispan.security.Security.doPrivileged(Security.java:76)

              at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:63)

              at org.infinispan.server.infinispan.SecurityActions.startCache(SecurityActions.java:135)

              at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:86)

              at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948) [jboss-msc-1.2.6.Final.jar:1.2.6.Final]

              at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881) [jboss-msc-1.2.6.Final.jar:1.2.6.Final]

              ... 3 more

      Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on vsd-kmurthy-set2-node1.mv.nuagenetworks.net

              at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:217)

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_111]

              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_111]

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_111]

              at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111]

              at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)

              ... 21 more