Replicated Cache - org.infinispan.util.concurrent.TimeoutException: ISPN000476

[nrmohta]

Hi All,

We are using infinispan replicated cache and observing below error in our performance test lab setup.

Infinispan version - 9.1.1 Final

Replicated Cache Configuration :

<replicated-cache name="defaultCache" mode="ASYNC" queue-size="500" queue-flush-interval="10000">

         <state-transfer await-initial-transfer="false"

            timeout="240000"

            enabled="true"

            chunk-size="10000"

         />

         <locking concurrency-level="1000" striping="true"

                    isolation="REPEATABLE_READ" write-skew="false"/>

      </replicated-cache>

Error Stacktrace :

org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 6250 from tsps-ha-win02-61896

    at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)

    at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1679)

    at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1327)

    at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1793)

    at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:282)

    at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:358)

    at org.infinispan.cache.impl.EncoderCache.put(EncoderCache.java:655)

    at com.bmc.truesight.platform.etl.mo.task.TSIMMOCRUDTask.process(TSIMMOCRUDTask.java:117)

    at com.bmc.truesight.api.qmgr.TSQueue.run(TSQueue.java:85)

    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

    at java.util.concurrent.FutureTask.run(FutureTask.java:266)

    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

    at java.lang.Thread.run(Thread.java:748)

Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 6250 from tsps-ha-win02-61896

    at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64)

    at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86)

    at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21)

    ... 6 common frames omitted

[suenda]

When you start your cache with await-initial-transfer="false" and if your cache has not yet finished the initial state-transfer, it simply waits for state-transfer to finish in background. Verify via JMX if the MBeans Cache ->StateTransferManger->rebalancingStatus = COMPLETE. If you get the error even after state-transfer is complete, try increasing the value of remote-timeout in your <replicated-cache /> configuration. The value of remote-timeout is "15000" milliseconds by default. Increase the timeout until you stop getting the error.