We are currently running jboss 3.2.6 clustered application of 4 nodes. There are two unix servers and two cluster jboss nodes on each. Let's name them node11, node12, node21 and node22.
We use singleton object, deployed via deploy-hassingleton. This singleton object is active and live only on Master node. And all other nodes simply use this master node service. Standard functionality
The problem is when cluster is starting, or current master node is failing and cluster needs new master node - it looks like nodes - node11, node12 are unable to serve as normal master node.
If Node11 (or node12) becomes master node - the singleton object works fine only on that node, but other nodes in the cluster do not see it.
There are such loglines in server.log
2013-08-28 09:13:16,123 WARN [org.jboss.ha.framework.interfaces.HAPartition.WorkflowPartition] javax.naming.NameNotFoundException: SingletonRMIAdaptor not bound
Strange, but this is not happening when master node becomes one of node21 or node22. It serves as normal master node, and singleton objects works fine, and all other nodes in cluster can see it.
Writing this post I almost answered myself, where may be the problem. It has to be something specific to this unix server, on which node11 and node12 are running. Maybe some ports are not open.
Clustering works fine (as much as I understand it), in server.log i can see, that all 4 nodes see each other.
Just I'm not shure yet, wat ports, network protocols are beeing used in clustering comunication, and/or how could i debug (find out) them. Can someone here help me?
Configuration of all 4 nodes are identical, I made all 4 nodes as clones of one jboss node.
IF some more specific information is needed - please ask and I will try to post it.