Why do we see state transfer timeout occasionally
richmayfield Dec 1, 2014 2:20 PMWe occasionally see the following error when recycling (restarting) nodes in our Infinispan cluster. All caches are configured with CacheMode.INVALIDATION_ASYNC.
org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:170)
at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)
at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)
at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:216)
at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:812)
at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:584)
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:539)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:416)
...
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache Instance_6623 on dev-64763
at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:216)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
... 38 more
Although we have a lot of caches (many 1000's) they are all invalidation caches and so I wouldn't expect there to be much, if any, information to transfer.
I have seen mention of setting <state-transfer timeout="..."/>, however looking at the code that handles configuration it doesn't look like invalidation cache configuration pays attention to <state-transfer />.
I have been seeing this occasionally with 7.0.0.Final. Moving to 7.0.2.Final just to be on the latest.
Wondering if anyone could explain whether or not state-transfer is even relevant here and/or why one might see this in a pure "invalidation" mode.