unusual clustering error causing jboss apps to freeze.
I have two jboss 4.0.4CR1 w/ejb3 servers clustered with session failover. They were running great for about 1 month. Then all of a sudden out of nowhere I get this clustering warning message on both machines and both of them freeze all of sudden. I had to kill both app servers and restart as a consequence.
Below is a sample of the messages from one of the machines, the other machine exhibits similar messages (twiddle and toodle are the two names of the machines).
2006-07-13 21:04:29,799 ERROR [org.jgroups.protocols.pbcast.GMS] [twiddle:36484] received view <= current view; discarding it (current vid: [twiddle:36484|6], new vid: [twiddle:36484|6])
2006-07-13 21:04:38,502 WARN [org.jgroups.protocols.pbcast.NAKACK] [twiddle:36488 (additional data: 19 bytes)] discarded message from non-member toodle:34338 (additional data: 19 bytes)
2006-07-13 21:04:38,503 WARN [org.jgroups.protocols.pbcast.NAKACK] [twiddle:36488 (additional data: 19 bytes)] discarded message from non-member toodle:34338 (additional data: 19 bytes)
2006-07-13 21:04:38,503 WARN [org.jgroups.protocols.pbcast.NAKACK] [twiddle:36488 (additional data: 19 bytes)] discarded message from non-member toodle:34338 (additional data: 19 bytes)
2006-07-13 21:04:55,795 WARN [org.jgroups.protocols.pbcast.NAKACK] [twiddle:36488 (additional data: 19 bytes)] discarded message from non-member toodle:34338 (additional data: 19 bytes)
2006-07-13 21:04:55,795 WARN [org.jgroups.protocols.pbcast.NAKACK] [twiddle:36488 (additional data: 19 bytes)] discarded message from non-member toodle:34338 (additional data: 19 bytes)
2006-07-13 21:04:55,796 WARN [org.jgroups.protocols.pbcast.NAKACK] [twiddle:36488 (additional data: 19 bytes)] discarded message from non-member toodle:34338 (additional data: 19 bytes)
2006-07-13 21:04:55,796 WARN [org.jgroups.protocols.pbcast.NAKACK] [twiddle:36488 (additional data: 19 bytes)] discarded message from non-member toodle:34338 (additional data: 19 bytes)
2006-07-13 21:05:06,509 WARN [org.jgroups.protocols.FD] I was suspected, but will not remove myself from membership (waiting for EXIT message)
2006-07-13 21:05:06,509 WARN [org.jgroups.protocols.pbcast.CoordGmsImpl] merge responses from subgroup coordinators <= 1 ([sender=twiddle:36488 (additional data: 19 bytes), view=[twiddle:36488 (additional data: 19 bytes)|4] [twiddle:36488 (additional data: 19 bytes)], digest=[twiddle:36488 (additional data: 19 bytes): [0 : 31]]). Cancelling merge
2006-07-13 21:05:06,520 WARN [org.jgroups.protocols.pbcast.GMS] checkSelfInclusion() failed, twiddle:36484 is not a member of view [toodle:34335|7] [toodle:34335]; discarding view
2006-07-13 21:05:06,520 WARN [org.jgroups.protocols.pbcast.GMS] I (twiddle:36484) am being shunned, will leave and rejoin group (prev_members are [twiddle:36484 toodle:34313 toodle:34327 toodle:34335 ])
2006-07-13 21:05:15,218 WARN [org.jgroups.protocols.pbcast.CoordGmsImpl] I am the coord and I'm being am suspected -- will probably leave shortly
2006-07-13 21:05:29,274 WARN [org.jgroups.protocols.FD] I was suspected, but will not remove myself from membership (waiting for EXIT message)