Hi, am seeing the following error when starting a new node in a cluster. The cache in question is REPL_SYNC. I read a few posts about this error but they don't seem to match our scenario, or they were to do with bugs since fixed.
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache active_rule on ip-172-31-9-161-1572
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
I am going to work around this by setting awaitInitialTransfer(false)... this cache has a backing store.
What is slightly confusing me about the exception is that '172-31-9-161' is the node that is starting. Is it in a deadlock waiting for itself? Or is this just information about the node having a problem getting the initial state?
Some further information...
- There are three nodes in the cluster
- This doesn't happen all the time. Normally new nodes are able to join no problem, but having got itself in this state, this node is consistently unable to join, timing out after 4 minutes with the same error
- The two healthy nodes were under load at the time I first tried to join the third node, but having removed the load, problem joining persists
- We are using mongo ping to coordinate cluster members (running in EC2)
- After this failure, the cluster is in a strange state. The cluster size is reported as 3, but am seeing NoClassDef errors both on the node in question and the other nodes (reporting the remote exception from cluster member). It's like the node is half in the cluster but didn't initialise properly. This is likely because my app is Spring-based and after the initial timeout the Spring context startup is abandoned, so there is other initialisation not happening. Still an odd error though (examples below)
2014-09-16 09:43:37,945 ERROR TimeScheduler3 | failed executing task UNICAST3: RetransmitTask (interval=500 ms)
Caused by: java.lang.ClassNotFoundException: org.jgroups.util.Table$Missing
... 8 more
2014-09-16 09:43:37,960 ERROR TCP | JGRP000030: ip-172-31-9-161-35145: failed handling incoming message: java.lang.NoClassDefFoundError: org/jgroups/protocols/TP$BatchHandler