Infinispan rebalance issue (9.0.1 and TCPPING)
auth.gabor Aug 21, 2015 7:53 AMI've installed the Wildfly 9.0.1.Final to my four VPS and configured the TCPPING based TCP stack:
<stack name="tcpping">
<transport type="TCP" socket-binding="jgroups-tcp"/><protocol type="TCPPING">
<property name="port_range">0</property>
<property name="initial_hosts">...[7600],...[7600],...[7600],...[7600]</property>
</protocol>
<protocol type="MERGE2"/>
<protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
<protocol type="FD"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="BARRIER"/>
<protocol type="pbcast.NAKACK"/>
<protocol type="UNICAST2"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS">
<property name="join_timeout">3000</property>
</protocol>
<protocol type="MFC"/>
<protocol type="FRAG2"/>
<protocol type="RSVP"/>
</stack>
It works (the HornetQ separately configured):
2015-08-21 10:15:40,580 INFO [org.hornetq.core.server] (Thread-22 (HornetQ-server-HornetQServerImpl::serverUUID=721d266c-471b-11e5-804f-2d915de25069-102213587)) HQ221027: Bridge ClusterConnectionBridge@520c2b01 [name=sf.my-cluster.2390a8bf-471b-11e5-9bf2-1f8985d0deeb, queue=QueueImpl[name=sf.my-cluster.2390a8bf-471b-11e5-9bf2-1f8985d0deeb, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=721d266c-471b-11e5-804f-2d915de25069]]@3299b324 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@520c2b01 [name=sf.my-cluster.2390a8bf-471b-11e5-9bf2-1f8985d0deeb, queue=QueueImpl[name=sf.my-cluster.2390a8bf-471b-11e5-9bf2-1f8985d0deeb, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=721d266c-471b-11e5-804f-2d915de25069]]@3299b324 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=http-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=8080&host=x-x-x-x&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@2123237246[nodeUUID=721d266c-471b-11e5-804f-2d915de25069, connector=TransportConfiguration(name=http-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=8080&host=x-x-x-x&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor, address=jms, server=HornetQServerImpl::serverUUID=721d266c-471b-11e5-804f-2d915de25069])) [initialConnectors=[TransportConfiguration(name=http-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=8080&host=x-x-x-x&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor], discoveryGroupConfiguration=null]] is connected
2015-08-21 10:15:42,331 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-11,ee,dc02-wild02) ISPN000094: Received new cluster view for channel web: [dc01-wild01|17] (4) [dc01-wild01, dc01-wild02, dc02-wild02, dc02-wild01]
2015-08-21 10:15:42,334 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-11,ee,dc02-wild02) ISPN000094: Received new cluster view for channel ejb: [dc01-wild01|17] (4) [dc01-wild01, dc01-wild02, dc02-wild02, dc02-wild01]
The topology is based on two DC and two node in each DC: dc01-wild01, dc01-wild02, dc02-wild01 and dc02-wild02
I've made a short network split between the two DC:
2015-08-21 00:27:39,801 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-13,ee,dc01-wild01) ISPN000094: Received new cluster view for channel web: [dc01-wild01|5] (2) [dc01-wild01, dc01-wild02]
2015-08-21 00:27:39,801 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-13,ee,dc01-wild01) ISPN000094: Received new cluster view for channel ejb: [dc01-wild01|5] (2) [dc01-wild01, dc01-wild02]
2015-08-21 00:28:01,636 WARN [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212037: Connection failure has been detected: HQ119014: Did not receive data from /dc02-wild01:50755. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
2015-08-21 00:28:01,641 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session f61066c0-476f-11e5-937d-51211e930346
2015-08-21 00:28:01,652 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session f61066c0-476f-11e5-937d-51211e930346
2015-08-21 00:28:01,676 INFO [org.hornetq.core.server] (hornetq-failure-check-thread) HQ221021: failed to remove connection
2015-08-21 00:28:01,674 WARN [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212037: Connection failure has been detected: HQ119014: Did not receive data from /dc02-wild02:50753. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
2015-08-21 00:28:05,678 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session e6b45da5-476f-11e5-bf0e-21b2d764fb60
2015-08-21 00:28:05,679 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session e6b45da5-476f-11e5-bf0e-21b2d764fb60
2015-08-21 00:28:05,683 INFO [org.hornetq.core.server] (hornetq-failure-check-thread) HQ221021: failed to remove connection
[...]
2015-08-21 00:28:39,769 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-15,ee,dc02-wild01) ISPN000093: Received new, MERGED cluster view for channel web: MergeView::[dc02-wild02|7] (2) [dc02-wild02, dc02-wild01], 1 subgroups: [dc02-wild02|6] (1) [dc02-wild02]
2015-08-21 00:28:39,769 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-15,ee,dc02-wild01) ISPN000093: Received new, MERGED cluster view for channel ejb: MergeView::[dc02-wild02|7] (2) [dc02-wild02, dc02-wild01], 1 subgroups: [dc02-wild02|6] (1) [dc02-wild02]
2015-08-21 00:28:35,774 WARN [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212037: Connection failure has been detected: HQ119014: Did not receive data from /dc01-wild01:39101. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
2015-08-21 00:28:35,776 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session e73bb549-476f-11e5-80eb-ab455268dbbc
2015-08-21 00:28:35,779 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session e73bb549-476f-11e5-80eb-ab455268dbbc
2015-08-21 00:28:35,803 INFO [org.hornetq.core.server] (hornetq-failure-check-thread) HQ221021: failed to remove connection
2015-08-21 00:28:35,803 WARN [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212037: Connection failure has been detected: HQ119014: Did not receive data from /dc01-wild02:44217. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
2015-08-21 00:28:35,804 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session e741cfed-476f-11e5-ba8e-d150e8aecec4
2015-08-21 00:28:35,804 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session e741cfed-476f-11e5-ba8e-d150e8aecec4
2015-08-21 00:28:35,807 INFO [org.hornetq.core.server] (hornetq-failure-check-thread) HQ221021: failed to remove connection
...and I've restored the network state between the two DC:
2015-08-21 00:30:08,666 INFO [org.hornetq.core.server] (Thread-9 (HornetQ-server-HornetQServerImpl::serverUUID=6380e9fd-471b-11e5-96ec-31025eb08071-1856560111)) HQ221027: Bridge ClusterConnectionBridge@6cdd7c57 [name=sf.my-cluster.2390a8bf-471b-11e5-9bf2-1f8985d0deeb, queue=QueueImpl[name=sf.my-cluster.2390a8bf-471b-11e5-9bf2-1f8985d0deeb, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=6380e9fd-471b-11e5-96ec-31025eb08071]]@621a8fbc targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@6cdd7c57 [name=sf.my-cluster.2390a8bf-471b-11e5-9bf2-1f8985d0deeb, queue=QueueImpl[name=sf.my-cluster.2390a8bf-471b-11e5-9bf2-1f8985d0deeb, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=6380e9fd-471b-11e5-96ec-31025eb08071]]@621a8fbc targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=http-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=8080&host=x-x-x-x&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@222674417[nodeUUID=6380e9fd-471b-11e5-96ec-31025eb08071, connector=TransportConfiguration(name=http-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=8080&host=x-x-x-x&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor, address=jms, server=HornetQServerImpl::serverUUID=6380e9fd-471b-11e5-96ec-31025eb08071])) [initialConnectors=[TransportConfiguration(name=http-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=8080&host=x-x-x-x&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor], discoveryGroupConfiguration=null]] is connected
2015-08-21 00:30:09,720 INFO [org.hornetq.core.server] (Thread-25 (HornetQ-server-HornetQServerImpl::serverUUID=6380e9fd-471b-11e5-96ec-31025eb08071-1856560111)) HQ221027: Bridge ClusterConnectionBridge@328f547d [name=sf.my-cluster.721d266c-471b-11e5-804f-2d915de25069, queue=QueueImpl[name=sf.my-cluster.721d266c-471b-11e5-804f-2d915de25069, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=6380e9fd-471b-11e5-96ec-31025eb08071]]@3c48ac5c targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@328f547d [name=sf.my-cluster.721d266c-471b-11e5-804f-2d915de25069, queue=QueueImpl[name=sf.my-cluster.721d266c-471b-11e5-804f-2d915de25069, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=6380e9fd-471b-11e5-96ec-31025eb08071]]@3c48ac5c targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=http-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=8080&host=x-x-x-x&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@222674417[nodeUUID=6380e9fd-471b-11e5-96ec-31025eb08071, connector=TransportConfiguration(name=http-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=8080&host=x-x-x-x&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor, address=jms, server=HornetQServerImpl::serverUUID=6380e9fd-471b-11e5-96ec-31025eb08071])) [initialConnectors=[TransportConfiguration(name=http-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=8080&host=x-x-x-x&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor], discoveryGroupConfiguration=null]] is connected
2015-08-21 00:30:27,181 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-8,ee,dc01-wild01) ISPN000093: Received new, MERGED cluster view for channel web: MergeView::[dc01-wild01|8] (4) [dc01-wild01, dc02-wild02, dc02-wild01, dc01-wild02], 2 subgroups: [dc02-wild02|7] (2) [dc02-wild02, dc02-wild01], [dc01-wild01|5] (2) [dc01-wild01, dc01-wild02]
2015-08-21 00:30:27,182 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-8,ee,dc01-wild01) ISPN000093: Received new, MERGED cluster view for channel ejb: MergeView::[dc01-wild01|8] (4) [dc01-wild01, dc02-wild02, dc02-wild01, dc01-wild02], 2 subgroups: [dc02-wild02|7] (2) [dc02-wild02, dc02-wild01], [dc01-wild01|5] (2) [dc01-wild01, dc01-wild02]
[...]
2015-08-21 00:30:07,957 INFO [org.hornetq.core.server] (Thread-18 (HornetQ-server-HornetQServerImpl::serverUUID=2390a8bf-471b-11e5-9bf2-1f8985d0deeb-53245283)) HQ221027: Bridge ClusterConnectionBridge@14c7f868 [name=sf.my-cluster.46d0dc0a-471b-11e5-a0d3-e33b3b454637, queue=QueueImpl[name=sf.my-cluster.46d0dc0a-471b-11e5-a0d3-e33b3b454637, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=2390a8bf-471b-11e5-9bf2-1f8985d0deeb]]@7f1602b targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@14c7f868 [name=sf.my-cluster.46d0dc0a-471b-11e5-a0d3-e33b3b454637, queue=QueueImpl[name=sf.my-cluster.46d0dc0a-471b-11e5-a0d3-e33b3b454637, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=2390a8bf-471b-11e5-9bf2-1f8985d0deeb]]@7f1602b targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=http-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=8080&host=x-x-x-x&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@521648698[nodeUUID=2390a8bf-471b-11e5-9bf2-1f8985d0deeb, connector=TransportConfiguration(name=http-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=8080&host=x-x-x-x&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor, address=jms, server=HornetQServerImpl::serverUUID=2390a8bf-471b-11e5-9bf2-1f8985d0deeb])) [initialConnectors=[TransportConfiguration(name=http-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=8080&host=x-x-x-x&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor], discoveryGroupConfiguration=null]] is connected
2015-08-21 00:30:08,945 INFO [org.hornetq.core.server] (Thread-15 (HornetQ-server-HornetQServerImpl::serverUUID=2390a8bf-471b-11e5-9bf2-1f8985d0deeb-53245283)) HQ221027: Bridge ClusterConnectionBridge@3bf1b6b [name=sf.my-cluster.6380e9fd-471b-11e5-96ec-31025eb08071, queue=QueueImpl[name=sf.my-cluster.6380e9fd-471b-11e5-96ec-31025eb08071, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=2390a8bf-471b-11e5-9bf2-1f8985d0deeb]]@1f981c97 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@3bf1b6b [name=sf.my-cluster.6380e9fd-471b-11e5-96ec-31025eb08071, queue=QueueImpl[name=sf.my-cluster.6380e9fd-471b-11e5-96ec-31025eb08071, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=2390a8bf-471b-11e5-9bf2-1f8985d0deeb]]@1f981c97 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=http-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=8080&host=x-x-x-x&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@521648698[nodeUUID=2390a8bf-471b-11e5-9bf2-1f8985d0deeb, connector=TransportConfiguration(name=http-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=8080&host=x-x-x-x&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor, address=jms, server=HornetQServerImpl::serverUUID=2390a8bf-471b-11e5-9bf2-1f8985d0deeb])) [initialConnectors=[TransportConfiguration(name=http-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=8080&host=x-x-x-x&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor], discoveryGroupConfiguration=null]] is connected
2015-08-21 00:30:27,315 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,ee,dc02-wild01) ISPN000093: Received new, MERGED cluster view for channel web: MergeView::[dc01-wild01|8] (4) [dc01-wild01, dc02-wild02, dc02-wild01, dc01-wild02], 2 subgroups: [dc02-wild02|7] (2) [dc02-wild02, dc02-wild01], [dc01-wild01|5] (2) [dc01-wild01, dc01-wild02]
2015-08-21 00:30:27,315 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,ee,dc02-wild01) ISPN000093: Received new, MERGED cluster view for channel ejb: MergeView::[dc01-wild01|8] (4) [dc01-wild01, dc02-wild02, dc02-wild01, dc01-wild02], 2 subgroups: [dc02-wild02|7] (2) [dc02-wild02, dc02-wild01], [dc01-wild01|5] (2) [dc01-wild01, dc01-wild02]
As you see, the HornetQ and the Wildfly cluster joined together...
...but the Infinispan not, some exception from the server.log of the nodes:
2015-08-21 00:27:39,802 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p3-t25) ISPN000197: Error updating cluster member list: org.infinispan.remoting.transport.jgroups.SuspectException: Suspected member: dc02-wild02
at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:78)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:586)
at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:402)
at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:393)
at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:309)
at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener$1.run(ClusterTopologyManagerImpl.java:590)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[...]
2015-08-21 00:27:59,371 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (transport-thread--p3-t3) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 8
at org.infinispan.statetransfer.StateTransferLockImpl.waitForTransactionData(StateTransferLockImpl.java:92)
at org.infinispan.interceptors.base.BaseStateTransferInterceptor.waitForTransactionData(BaseStateTransferInterceptor.java:96)
at org.infinispan.statetransfer.StateTransferInterceptor.handleTxWriteCommand(StateTransferInterceptor.java:278)
at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:247)
at org.infinispan.statetransfer.StateTransferInterceptor.visitRemoveCommand(StateTransferInterceptor.java:123)
at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.CacheMgmtInterceptor.visitRemoveCommand(CacheMgmtInterceptor.java:209)
at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:49)
at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1617)
at org.infinispan.cache.impl.CacheImpl.removeInternal(CacheImpl.java:579)
at org.infinispan.cache.impl.CacheImpl.remove(CacheImpl.java:572)
at org.infinispan.cache.impl.DecoratedCache.remove(DecoratedCache.java:442)
at org.infinispan.cache.impl.AbstractDelegatingCache.remove(AbstractDelegatingCache.java:297)
at org.wildfly.clustering.server.registry.CacheRegistry.topologyChanged(CacheRegistry.java:152)
at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:286)
at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:22)
at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:309)
at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1212)
at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1170)
at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1135)
at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyTopologyChanged(CacheNotifierImpl.java:590)
at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:201)
at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:45)
at org.infinispan.statetransfer.StateTransferManagerImpl$1.updateConsistentHash(StateTransferManagerImpl.java:113)
at org.infinispan.topology.LocalTopologyManagerImpl.resetLocalTopologyBeforeRebalance(LocalTopologyManagerImpl.java:333)
at org.infinispan.topology.LocalTopologyManagerImpl.doHandleRebalance(LocalTopologyManagerImpl.java:413)
at org.infinispan.topology.LocalTopologyManagerImpl$3.run(LocalTopologyManagerImpl.java:382)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.infinispan.executors.SemaphoreCompletionService$QueueingTask.runInternal(SemaphoreCompletionService.java:173)
at org.infinispan.executors.SemaphoreCompletionService$QueueingTask.run(SemaphoreCompletionService.java:151)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[...]
2015-08-21 00:27:59,446 ERROR [org.infinispan.topology.LocalTopologyManagerImpl] (transport-thread--p8-t13) ISPN000367: There was an issue with topology update for topology: 9: org.infinispan.commons.CacheListenerException: I
SPN000280: Caught exception [org.infinispan.util.concurrent.TimeoutException] while invoking method [public void org.wildfly.clustering.server.registry.CacheRegistry.topologyChanged(org.infinispan.notifications.cachelistener.
event.TopologyChangedEvent)] on listener instance: org.wildfly.clustering.server.registry.CacheRegistryFactory$1@7014c85
at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:291)
at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:22)
at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:309)
at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1212)
at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1170)
at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1135)
at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyTopologyChanged(CacheNotifierImpl.java:590)
at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:201)
at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:45)
at org.infinispan.statetransfer.StateTransferManagerImpl$1.updateConsistentHash(StateTransferManagerImpl.java:113)
at org.infinispan.topology.LocalTopologyManagerImpl.doHandleTopologyUpdate(LocalTopologyManagerImpl.java:285)
at org.infinispan.topology.LocalTopologyManagerImpl$1.run(LocalTopologyManagerImpl.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.infinispan.executors.SemaphoreCompletionService$QueueingTask.runInternal(SemaphoreCompletionService.java:173)
at org.infinispan.executors.SemaphoreCompletionService$QueueingTask.run(SemaphoreCompletionService.java:151)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 8
at org.infinispan.statetransfer.StateTransferLockImpl.waitForTransactionData(StateTransferLockImpl.java:92)
at org.infinispan.interceptors.base.BaseStateTransferInterceptor.waitForTransactionData(BaseStateTransferInterceptor.java:96)
at org.infinispan.statetransfer.StateTransferInterceptor.handleTxWriteCommand(StateTransferInterceptor.java:278)
at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:247)
at org.infinispan.statetransfer.StateTransferInterceptor.visitRemoveCommand(StateTransferInterceptor.java:123)
at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.CacheMgmtInterceptor.visitRemoveCommand(CacheMgmtInterceptor.java:209)
at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:49)
at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1617)
at org.infinispan.cache.impl.CacheImpl.removeInternal(CacheImpl.java:579)
at org.infinispan.cache.impl.CacheImpl.remove(CacheImpl.java:572)
at org.infinispan.cache.impl.DecoratedCache.remove(DecoratedCache.java:442)
at org.infinispan.cache.impl.AbstractDelegatingCache.remove(AbstractDelegatingCache.java:297)
at org.wildfly.clustering.server.registry.CacheRegistry.topologyChanged(CacheRegistry.java:152)
at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:286)
... 18 more
Suppressed: org.infinispan.commons.CacheException: javax.transaction.RollbackException: Transaction marked as rollback only.
at org.wildfly.clustering.ee.infinispan.ActiveTransactionBatch.close(ActiveTransactionBatch.java:50)
at org.wildfly.clustering.server.registry.CacheRegistry.topologyChanged(CacheRegistry.java:157)
... 22 more
Caused by: javax.transaction.RollbackException: Transaction marked as rollback only.
at org.infinispan.transaction.tm.DummyTransaction.setRollbackOnly(DummyTransaction.java:148)
at org.infinispan.interceptors.InvocationContextInterceptor.markTxForRollbackAndRethrow(InvocationContextInterceptor.java:163)
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:49)
at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1617)
at org.infinispan.cache.impl.CacheImpl.removeInternal(CacheImpl.java:579)
at org.infinispan.cache.impl.CacheImpl.remove(CacheImpl.java:572)
at org.infinispan.cache.impl.DecoratedCache.remove(DecoratedCache.java:442)
at org.infinispan.cache.impl.AbstractDelegatingCache.remove(AbstractDelegatingCache.java:297)
at org.wildfly.clustering.server.registry.CacheRegistry.topologyChanged(CacheRegistry.java:152)
... 22 more
[...]
2015-08-21 00:30:33,189 ERROR [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p4-t7) ISPN000196: Failed to recover cluster state after the current node became the coordinator: java.util.concurrent.TimeoutException
at java.util.concurrent.FutureTask.get(FutureTask.java:205)
at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterSync(ClusterTopologyManagerImpl.java:473)
at org.infinispan.topology.ClusterTopologyManagerImpl.recoverClusterStatus(ClusterTopologyManagerImpl.java:350)
at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:286)
at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener$1.run(ClusterTopologyManagerImpl.java:590)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2015-08-21 00:30:42,262 WARN [org.infinispan.partitionhandling.impl.PreferAvailabilityStrategy] (transport-thread--p8-t20) ISPN000313: Cache dist lost data because of abrupt leavers [dc01-wild01, dc01-wild02]
[...]
2015-08-21 00:30:42,511 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p6-t26) ISPN000071: Caught exception when handling command StateRequestCommand{cache=dist, origin=dc01-wild02, type=GET_TRANSACTIONS, topologyId=8, segments=[32, 67, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 23, 25, 28, 30]}: java.lang.IllegalArgumentException: Node dc01-wild01 is not a member
at org.infinispan.distribution.ch.impl.DefaultConsistentHash.getSegmentsForOwner(DefaultConsistentHash.java:115)
at org.infinispan.distribution.group.GroupingConsistentHash.getSegmentsForOwner(GroupingConsistentHash.java:67)
at org.infinispan.statetransfer.StateProviderImpl.getTransactionsForSegments(StateProviderImpl.java:163)
at org.infinispan.statetransfer.StateRequestCommand.perform(StateRequestCommand.java:67)
at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:85)
at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[...]
2015-08-21 00:28:41,861 WARN [org.infinispan.statetransfer.StateConsumerImpl] (transport-thread--p5-t2) ISPN000209: Failed to retrieve transactions for segments [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 23, 25, 28, 30, 32, 67] of cache dist from node dc01-wild01: org.infinispan.util.concurrent.TimeoutException: Node dc01-wild01 timed out
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:248)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:561)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:287)
at org.infinispan.statetransfer.StateConsumerImpl.getTransactions(StateConsumerImpl.java:848)
at org.infinispan.statetransfer.StateConsumerImpl.requestTransactions(StateConsumerImpl.java:767)
at org.infinispan.statetransfer.StateConsumerImpl.addTransfers(StateConsumerImpl.java:711)
at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:369)
at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:198)
at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:45)
at org.infinispan.statetransfer.StateTransferManagerImpl$1.rebalance(StateTransferManagerImpl.java:118)
at org.infinispan.topology.LocalTopologyManagerImpl.doHandleRebalance(LocalTopologyManagerImpl.java:420)
at org.infinispan.topology.LocalTopologyManagerImpl$3.run(LocalTopologyManagerImpl.java:382)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.infinispan.executors.SemaphoreCompletionService$QueueingTask.runInternal(SemaphoreCompletionService.java:173)
at org.infinispan.executors.SemaphoreCompletionService$QueueingTask.run(SemaphoreCompletionService.java:151)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.jgroups.TimeoutException: timeout waiting for response from dc01-wild01, request: org.jgroups.blocks.UnicastRequest@ce4af61, req_id=8688, mode=GET_ALL, target=dc01-wild01
at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:427)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:433)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:241)
... 18 more
[...]
2015-08-21 00:28:58,324 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (SessionExpirationScheduler - 1) ISPN000136: Execution error: java.lang.StackOverflowError
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.jboss.logging.Logger$1.run(Logger.java:2544)
at java.security.AccessController.doPrivileged(Native Method)
at org.jboss.logging.Logger.getMessageLogger(Logger.java:2529)
at org.jboss.logging.Logger.getMessageLogger(Logger.java:2516)
at org.infinispan.util.logging.LogFactory.getLog(LogFactory.java:17)
at org.infinispan.commands.remote.recovery.TxCompletionNotificationCommand.<clinit>(TxCompletionNotificationCommand.java:26)
at org.infinispan.commands.CommandsFactoryImpl.buildTxCompletionNotificationCommand(CommandsFactoryImpl.java:542)
at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.releaseLocksOnFailureBeforePrepare(PessimisticLockingInterceptor.java:252)
at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataWriteCommand(PessimisticLockingInterceptor.java:138)
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitRemoveCommand(AbstractLockingInterceptor.java:65)
at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:49)
at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58)
[...]
2015-08-21 00:28:58,426 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (SessionExpirationScheduler - 1) ISPN000136: Execution error: java.lang.NoClassDefFoundError: Could not initialize class org.infinispan.commands.remote.recovery.TxCompletionNotificationCommand
at org.infinispan.commands.CommandsFactoryImpl.buildTxCompletionNotificationCommand(CommandsFactoryImpl.java:542)
at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.releaseLocksOnFailureBeforePrepare(PessimisticLockingInterceptor.java:252)
at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataWriteCommand(PessimisticLockingInterceptor.java:138)
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitRemoveCommand(AbstractLockingInterceptor.java:65)
at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:49)
at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:367)
at org.infinispan.interceptors.TxInterceptor.visitRemoveCommand(TxInterceptor.java:273)
at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
[...]
2015-08-21 00:28:39,773 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-16) ISPN000136: Execution error: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from dc02-wild02,
see cause for remote stack trace
at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:46)
at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:71)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:586)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:287)
at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:185)
at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:174)
at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:174)
at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:174)
at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.acquireRemoteIfNeeded(PessimisticLockingInterceptor.java:238)
at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataWriteCommand(PessimisticLockingInterceptor.java:128)
at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitPutKeyValueCommand(AbstractTxLockingInterceptor.java:68)
[...]
2015-08-21 00:28:42,482 WARN [org.infinispan.transaction.impl.TransactionTable] (TxCleanupService,gacivs-frontend-war-0.0.22-SNAPSHOT.war,dc02-wild01) ISPN000326: Remote transaction GlobalTransaction:<dc02-wild02>:6506:remote timed out. Rolling back after 67032 ms
2015-08-21 00:28:42,483 WARN [org.infinispan.transaction.impl.TransactionTable] (TxCleanupService,gacivs-frontend-war-0.0.22-SNAPSHOT.war,dc02-wild01) ISPN000326: Remote transaction GlobalTransaction:<dc01-wild01>:10945:remote timed out. Rolling back after 66123 ms
[...]
2015-08-21 00:29:41,823 WARN [org.infinispan.statetransfer.StateConsumerImpl] (transport-thread--p8-t21) ISPN000209: Failed to retrieve transactions for segments [13, 14, 15, 16, 27, 28, 29, 30, 31, 32, 33, 54, 55, 67, 68, 70, 72] of cache dist from node dc02-wild01: org.infinispan.util.concurrent.TimeoutException: Node dc02-wild01 timed out
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:248)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:561)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:287)
at org.infinispan.statetransfer.StateConsumerImpl.getTransactions(StateConsumerImpl.java:848)
at org.infinispan.statetransfer.StateConsumerImpl.requestTransactions(StateConsumerImpl.java:767)
at org.infinispan.statetransfer.StateConsumerImpl.addTransfers(StateConsumerImpl.java:711)
at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:369)
at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:198)
at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:45)
at org.infinispan.statetransfer.StateTransferManagerImpl$1.rebalance(StateTransferManagerImpl.java:118)
at org.infinispan.topology.LocalTopologyManagerImpl.doHandleRebalance(LocalTopologyManagerImpl.java:420)
at org.infinispan.topology.LocalTopologyManagerImpl$3.run(LocalTopologyManagerImpl.java:382)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.infinispan.executors.SemaphoreCompletionService$QueueingTask.runInternal(SemaphoreCompletionService.java:173)
at org.infinispan.executors.SemaphoreCompletionService$QueueingTask.run(SemaphoreCompletionService.java:151)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.jgroups.TimeoutException: timeout waiting for response from dc02-wild01, request: org.jgroups.blocks.UnicastRequest@37ef53c9, req_id=5727, mode=GET_ALL, target=dc02-wild01
at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:427)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:433)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:241)
... 18 more
Every node logged these various exceptions ~2GByte/hour speed... :/
Any idea?
P.S.: The cache coordinator node and the cluster-wide rebalance of the cache is a new feature? It is works for you?
2015-08-21 11:45:42,426 INFO [org.infinispan.CLUSTER] (remote-thread--p9-t12) ISPN000310: Starting cluster-wide rebalance for cache gacivs-webgl-war-0.0.22-SNAPSHOT.war, topology CacheTopology{id=6, rebalanceId=3, currentCH=DefaultConsistentHash{ns=80, owners = (3)[dc01-wild01: 27+27, dc01-wild02: 27+26, dc02-wild01: 26+27]}, pendingCH=DefaultConsistentHash{ns=80, owners = (4)[dc01-wild01: 20+20, dc01-wild02: 20+20, dc02-wild01: 20+20, dc02-wild02: 20+20]}, unionCH=null, actualMembers=[dc01-wild01, dc01-wild02, dc02-wild01, dc02-wild02]}
2015-08-21 11:45:42,430 INFO [org.infinispan.CLUSTER] (remote-thread--p4-t1) ISPN000310: Starting cluster-wide rebalance for cache dist, topology CacheTopology{id=6, rebalanceId=3, currentCH=DefaultConsistentHash{ns=80, owners = (3)[dc01-wild01: 27+27, dc01-wild02: 27+26, dc02-wild01: 26+27]}, pendingCH=DefaultConsistentHash{ns=80, owners = (4)[dc01-wild01: 20+20, dc01-wild02: 20+20, dc02-wild01: 20+20, dc02-wild02: 20+20]}, unionCH=null, actualMembers=[dc01-wild01, dc01-wild02, dc02-wild01, dc02-wild02]}
2015-08-21 11:45:42,445 INFO [org.infinispan.CLUSTER] (remote-thread--p9-t17) ISPN000310: Starting cluster-wide rebalance for cache dist, topology CacheTopology{id=6, rebalanceId=3, currentCH=DefaultConsistentHash{ns=80, owners = (3)[dc01-wild01: 27+27, dc01-wild02: 27+26, dc02-wild01: 26+27]}, pendingCH=DefaultConsistentHash{ns=80, owners = (4)[dc01-wild01: 20+20, dc01-wild02: 20+20, dc02-wild01: 20+20, dc02-wild02: 20+20]}, unionCH=null, actualMembers=[dc01-wild01, dc01-wild02, dc02-wild01, dc02-wild02]}
2015-08-21 11:45:42,449 INFO [org.infinispan.CLUSTER] (remote-thread--p9-t16) ISPN000310: Starting cluster-wide rebalance for cache gacivs-frontend-war-0.0.22-SNAPSHOT.war, topology CacheTopology{id=6, rebalanceId=3, currentCH=DefaultConsistentHash{ns=80, owners = (3)[dc01-wild01: 27+27, dc01-wild02: 27+26, dc02-wild01: 26+27]}, pendingCH=DefaultConsistentHash{ns=80, owners = (4)[dc01-wild01: 20+20, dc01-wild02: 20+20, dc02-wild01: 20+20, dc02-wild02: 20+20]}, unionCH=null, actualMembers=[dc01-wild01, dc01-wild02, dc02-wild01, dc02-wild02]}
2015-08-21 11:45:42,453 INFO [org.infinispan.CLUSTER] (remote-thread--p9-t12) ISPN000310: Starting cluster-wide rebalance for cache gacivs-backend-services-0.0.22-SNAPSHOT.war, topology CacheTopology{id=6, rebalanceId=3, currentCH=DefaultConsistentHash{ns=80, owners = (3)[dc01-wild01: 27+27, dc01-wild02: 27+26, dc02-wild01: 26+27]}, pendingCH=DefaultConsistentHash{ns=80, owners = (4)[dc01-wild01: 20+20, dc01-wild02: 20+20, dc02-wild01: 20+20, dc02-wild02: 20+20]}, unionCH=null, actualMembers=[dc01-wild01, dc01-wild02, dc02-wild01, dc02-wild02]}
2015-08-21 11:45:42,471 INFO [org.infinispan.CLUSTER] (remote-thread--p9-t19) ISPN000310: Starting cluster-wide rebalance for cache gacivs-backend-dao-services-0.0.22-SNAPSHOT.war, topology CacheTopology{id=6, rebalanceId=3, currentCH=DefaultConsistentHash{ns=80, owners = (3)[dc01-wild01: 27+27, dc01-wild02: 27+26, dc02-wild01: 26+27]}, pendingCH=DefaultConsistentHash{ns=80, owners = (4)[dc01-wild01: 20+20, dc01-wild02: 20+20, dc02-wild01: 20+20, dc02-wild02: 20+20]}, unionCH=null, actualMembers=[dc01-wild01, dc01-wild02, dc02-wild01, dc02-wild02]}
2015-08-21 11:45:43,320 INFO [org.infinispan.CLUSTER] (remote-thread--p9-t19) ISPN000336: Finished cluster-wide rebalance for cache gacivs-webgl-war-0.0.22-SNAPSHOT.war, topology id = 6
2015-08-21 11:45:43,375 INFO [org.infinispan.CLUSTER] (remote-thread--p9-t19) ISPN000336: Finished cluster-wide rebalance for cache gacivs-backend-services-0.0.22-SNAPSHOT.war, topology id = 6
2015-08-21 11:45:43,378 INFO [org.infinispan.CLUSTER] (remote-thread--p9-t18) ISPN000336: Finished cluster-wide rebalance for cache gacivs-frontend-war-0.0.22-SNAPSHOT.war, topology id = 6
2015-08-21 11:45:43,437 INFO [org.infinispan.CLUSTER] (remote-thread--p9-t18) ISPN000336: Finished cluster-wide rebalance for cache dist, topology id = 6
2015-08-21 11:45:43,453 INFO [org.infinispan.CLUSTER] (remote-thread--p4-t5) ISPN000336: Finished cluster-wide rebalance for cache dist, topology id = 6
2015-08-21 11:45:43,464 INFO [org.infinispan.CLUSTER] (remote-thread--p9-t18) ISPN000336: Finished cluster-wide rebalance for cache gacivs-backend-dao-services-0.0.22-SNAPSHOT.war, topology id = 6
Sorry for the TL;DR post...