JBOSS EAP5.2 whole cluster hanged up

[sabir_mustafa]

Hi Everyone:

A few days back i had faced a strange problem in existing infrastructure running JBOSS EAP5.2 and having 12 JBOSS instances in a single cluster spanning on 06 machines . The whole application stopped responding. While investigating "server.log" file following message was appearing

5 WARN  [NAKACK] 192.168.3.10:55200] discarded message from non-member 192.168.3.14:55200, my view is [192.168.3.2:55200|12] [192.168.3.2:55200, 192.168.3.10:55200, 192.168.3.12:55200, 192.168.3.13:55200, 192.168.3.15:55200, 192.168.3.16:55200, 192.168.3.17:55200, 192.168.3.18:55200, 192.168.3.19:55200, 192.168.3.20:55200, 192.168.3.21:55200]

It was strange that only one node went out of cluster but the whole application stopped responding. first i just stopped the JBOSS instance on the affected machine and application resumed its services normally. Then started back the affected instance and everything became normal.

Anyone, please guide me how to avoid such problems in future, where shall i look for rectification.

Thanks

[ctomc]

Hey,

it would be best to create customer case in Red Hat Customer Portal so you will be able to get prompt help and support for your production environment.

[sabir_mustafa]

Hi Tomaz,

I am already in process with RedHat. The purpose of bringing it to community is to discuss it more in-depth and try to get more than one possible replies to handle such errors.

[ctomc]

Sure, but this forum is more for EAP 6.x so I am not sure how many people that use 5.x is hanging around here.

I am saying this more as heads up on why you might not get lots of replies.

[sabir_mustafa]

Do you have any related material on this specific topic. I mean the cluster troubleshooting in EAP5.

[wdfink]

moved to the cluster forum

[wdfink]

Sabir,

you might have a look to the JGroups documentation at www.jgroups.org

There you might find some hints and also troubleshooting sections.