Hello,

We create a two-node Infinispan cluster on centOS with version 9.1.4 of Infinispan. We use the clustered.xml with some customized settings for offset and multicast -IP. After a restart the cache cluster works a few minutes, then we get more and more issues with "Timeout waiting for response" But the appear and disappear of nodes is visible in the logs and in the admin interface. I have no idea what do I wrong. Here some error messages, the config will also be attached to this article.

2018-02-26 13:15:11,848 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p3-t1) ISPN000136: Error executing command PutKeyValueCommand, writing keys [WrappedByteArray{bytes=[B0x033E414C55434545..[68], hashCode=280211916}]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 518 from wwhelapp0115
 at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:163)
 at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86)
 at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)

or

2018-02-26 13:14:56,602 WARN  [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (timeout-thread--p3-t1) ISPN000071: Caught exception when handling command SingleRpcCommand{cacheName='helios', command=PutKeyValueCommand{key=WrappedByteArray{bytes=[B0x033E414C55434545..[68], hashCode=1308593447}, value=WrappedByteArray{bytes=[B0x0342021944001CC8..[1886332], hashCode=0}, flags=[IGNORE_RETURN_VALUES], commandInvocationId=CommandInvocation:wwhelapp0115:1036, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=86400000, maxIdle=28800000, version=NumericVersion{version=281479271678845}}, successful=true, topologyId=5}}: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 517 from wwhelapp0115
 at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:163)
 at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86)
 at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)

2018-02-26 14:39:12,590 WARN  [org.infinispan.expiration.impl.ClusterExpirationManager] (pool-4-thread-1) ISPN000026: Caught exception purging data container!: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 1008 from wwhelapp0116
 at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
 at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1661)
 at org.infinispan.cache.impl.CacheImpl.removeExpired(CacheImpl.java:654)
 at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.removeExpired(AbstractDelegatingAdvancedCache.java:383)
 at org.infinispan.cache.impl.EncoderCache.removeExpired(EncoderCache.java:486)
 at org.infinispan.expiration.impl.ClusterExpirationManager.removeExpired(ClusterExpirationManager.java:154)
 at org.infinispan.expiration.impl.ClusterExpirationManager.lambda$handleLifespanExpireEntry$0(ClusterExpirationManager.java:119)
 at org.infinispan.expiration.impl.ClusterExpirationManager.handleLifespanExpireEntry(ClusterExpirationManager.java:125)
 at org.infinispan.expiration.impl.ClusterExpirationManager.processExpiration(ClusterExpirationManager.java:91)
 at org.infinispan.expiration.impl.ExpirationManagerImpl$ScheduledTask.run(ExpirationManagerImpl.java:232)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)
Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 1008 from wwhelapp0116
 at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:64)
 at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86)
 at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
 ... 3 more