[brian.stansberry]

Not quite correct. FD doesn't "detect" a member, it validates that a known member is still alive. After "max_tries" heartbeat messages without a response, it initiates the process that leads to GMS excluding the non-responding member from the group. The fact you have shun="true" in both FD and GMS means that if that non-responding member attempts to communicate with the group again without going through the regular join process that happens when a channel connects, it will detect that it has been excluded and it will automatically reconnect.

Bottom line, FD shouldn't mean that a node that is functional gets kicked out of the group and can never participate again.

[mohitanchlia]

I think there is a problem with that process, and my understanding was that they introduced VIEW_SYNC to overcome that process. I am eager to hear jgroups response about VIEW_SYNC

[mohitanchlia]

Just checking if you heard from jgroups about VIEW_SYNC.

I just had one more question, in one of the previous replies you mentioned that HAController tells the cluster that it's there when it comes up. My understanding was that HAPartition notifies HAController and then HAController based on the Election policy notifies BArrierController. I thought HAController never talks to cluster, and only HAPartition talks to cluster and rest is done through notification mechanism. I am sure I am misunderstanding this from the clustering manual that I read.

[brian.stansberry]

I just pinged the JGroups folks again. You can also try the jgroups user mail list at https://lists.sourceforge.net/lists/listinfo/javagroups-users .

Re: HASingletonController, first thing to understand is that who the master is is a function of what nodes in the cluster have that HASingletonController deployed. There is the overall cluster view, which more people are familiar with, and then something that I call the "service view", which reflects what nodes in the cluster have a particular service deployed.

So, say you have a 4 node cluster and the view for the cluster is {A, B, C, D}. But the "Foo" service's HASingletonController is only deployed on 2 nodes. So it's service view is {C, D}. The "Bar" service's HASingletonController is deployed on a different 2 nodes, so it's service view is {A, B}.

In this example the master for the "Foo" service is C and the master for "Bar" is A.

A service view is always a subset of the overall cluster view. If a node crashes, it's removed from all service views.

How all this works behind the scenes is an HASingletonController uses a component of the HAPartition known as the DistributedReplicantManager. The DRM basically maintains the service views. When for example the "Foo" service HASingletonController is deployed on node D, it registers itself with the DRM by invoking the DRM add method. DRM in turn invokes a method on the HAPartition which does a group RPC telling the other DRMs in the cluster that "Foo" is now available on D. Those DRMs in turn issue a notification to the "Foo" HASingletonController on their nodes telling them that the service view for the "Foo" service has changed, and now includes "D". From that they can determine who the master is.

[mohitanchlia]

Such a great explanation. Why don't I get this from docs :) I know it takes so much keep up with the docs.

So if I understand correctly, in above example if for some reason election policy choses Node D as master then the view will be {D,C} instead of {C,D}. Is there a way in JMX Console to see this kind of view.

Also under the hoods does jgroup use NACKAK to do this kind of communication. Also are there different pings and FD to check Overall Cluster view and Service view.

Thanks again for educating me. It's hard to work with clusters when you don't know what exactly is going on.

[belaban]

VIEW_SYNC is discussed in http://javagroups.cvs.sourceforge.net/viewvc/javagroups/JGroups/doc/design/ReliableViewInstallation.txt?revision=1.1&view=markup.

To ask JGroups-related questions, your better bet is the JGroups mailing list.

[brian.stansberry]

"mohitanchlia" wrote:
Such a great explanation. Why don't I get this from docs :) I know it takes so much keep up with the docs.

http://www.jboss.org/file-access/default/members/jbossas/freezone/docs/Clustering_Guide/4/html/ch05s11s04.html is pretty close. :-) It lacks some of the implementation details of my post above; I'm currently redoing this guide for AS 5 so will think about adding that. Always a bit of a tradeoff; too much implementation detail and the concepts get lost, plus the implementation detail is changeable. And it's OSS, so the source is available.

So if I understand correctly, in above example if for some reason election policy choses Node D as master then the view will be {D,C} instead of {C,D}. Is there a way in JMX Console to see this kind of view.

Who the master is does not affect the service view; it remains {C, D} even if the election policy says D is the master.

Re: getting JMX Console info about the service view, you can access service view info via the jboss:service=DistributedReplicantManager,partitionName=DefaultPartition mbean. (Substitute your partition name for DefaultPartition if you configured something else, e.g. via the -g startup option). To use the mbean you need to know the name under which your service is registered. The "AllServices" mbean attribute lists them as a comma-delimited string. For an HASingletonController, by default it will be the the ObjectName of the controller mbean.

Once you know the service name you can paste it as a param to the lookupReplicantsNodeNames() operation. The result will be the service view in string form.

The DRM mbean also exposes a "isMasterReplica()" operation to which you pass the service name. Be cautious with this; it's a legacy operation from the days when the master was always the first node in the service view; all it does is tell you whether the current node is first in the service view.

You can go to the mbean for your HASingletonController itself; it will tell you if it is the master.

Also under the hoods does jgroup use NACKAK to do this kind of communication. Also are there different pings and FD to check Overall Cluster view and Service view.

Service views are maintained at a higher level via DRM telling HAPartition to do group RPCs. HAPartition uses JGroups to do that RPC. NAKACK comes into play in the sense that it ensures all members of the cluster receive the RPC and that they receive it in the correct order.

FD doesn't impact a service view directly, but it does have a big indirect impact, because a node can't be in a service view if it isn't also in the overall view. If JGroups suspects a node (e.g. due to not receiving FD heartbeat responses), the DRM gets a callback and that node is removed from all service views, and services interested in each service view get a callback telling them the service view changed.

Thanks again for educating me. It's hard to work with clusters when you don't know what exactly is going on.

Your best bet to understand in detail is to look at source code from http://anonsvn.jboss.org/repos/jbossas. Find the release you want under "tags"; most of the stuff relevant to this thread is in the "cluster" module under the root for the release.