It is suspicious that other nodes are joining the cluster fine except for one node. I would double-check the node system configuration, network/routing, firewall configuration, etc. Or is that failing node different every time?
As per the log, Node4 trying to join with Node1
In Node4 we are seeing following retry failures infinitely
2013-01-25 18:39:18,688 59144 WARN [org.jgroups.protocols.pbcast.GMS] (main:) join(10.0.215.4:55200) sent to 10.0.215.1:55200 timed out (after 3000 ms), retrying
at the same time in Node1 flush got failed
2013-01-25 18:46:58,479 905434 WARN [org.jgroups.protocols.pbcast.GMS] (ViewHandler,BNW_PROD_CONTROLLER-HAPartitionCache,10.0.215.1:55200:) GMS flush by coordinator at 10.0.215.1:55200 failed
The current JBOSS version we are using is JBOSS-5.1.1
It seems that this issue related to JIRA TICKET-
Thanks in advance.