HASingleton and partition merges

[brian.stansberry]

Discussion related to http://jira.jboss.com/jira/browse/JBAS-4229.

The way a clustered service is aware that a 'split brain' situation has occurred and is now healed is by receiving a MergeView from JGroups. Problem here is that HASingletonSupport does not receive view change notifications from JGroups -- it receives service topology change notifications from DistributedReplicantManager. (The topology of a service can change for reasons other than view changes; e.g. undeploy of the service on a node.) The topology change notification does not include any information as to whether the change was due to a merge.

In 4.x this can be somewhat shakily resolved by having the topology change callback do some extra work to try to figure out from HAPartition whether a MergeView has occurred. That is, register with the HAPartition for membership change events and track whether they are due to merges.

I say shakily resolved because the topology change notifications and the view change notifications come on separate threads with no guaranteed ordering. If numerous events are received in short order, the potential to have mis-correlation between topology changes and view changes certainly exists.

In 5.x a cleaner solution is possible, since in a major version I can change the DRM.replicantListener interface. I can add a boolean flag to the callback so the DRM can directly notify the singleton that the change was due to a merge.

[brian.stansberry]

The problem of ordering notifications can likely be solved.

Another issue is whether this kind of restart-on-merge behavior should be configurable or even turned on by default. HA-JMS needs this; for other use cases having the singleton restart may be undesirable. This is particularly an issue with the HASingletonDeployer mbean, since a restart there means redeploying everything in deploy-hasingleton, a potentially lengthy process.

I've no problem with adding a config flag for this, although that complicates backporting it for CP releases.

[galder.zamarreno]

I think it needs configuring too but I would leave it as off by default so that it mantains previous 4.0.x behaivour and then customers can set it to true for recommended services, such as HA-JMS.

What complications can you see in backporting this? The patch instructions could explain the flag and give the recommendations.

[brian.stansberry]

The complication with backporting is:

1) New config flag == change to mbean interface
2) Change to interface is illegal in a CP, hence have to create a new interface and new probably a trivial new class that exposes it (I'm too lazy to go beyond a StandardMBean approach.
3) Don't want to change deploy-hasingleton-service.xml to use this new class/interface, as that's too big a change for a CP (changing config file could break people who've already altered it.)

All this is resolvable, it's just extra hassle. And the result is that a fix will be in the codebase, but to take advantage of it users will have to make manual configuration changes.


(BTW, users can't set the flag to true for HA-JMS per se. They can set it to true for the jboss.ha:service=HASingletonDeployer service, which in essence means setting it to true for the entire deploy-hasingleton directory. If they wanted the behavior only for HA-JMS and had other deploy-hasingleton content, they'd have to separate the content into another dir and create a parallel HASingletonController to deploy that dir.)

[adrian.brock]

I think it needs configuring too but I would leave it as off by default so that it mantains previous 4.0.x behaivour and then customers can set it to true for recommended services, such as HA-JMS.

What complications can you see in backporting this? The patch instructions could explain the flag and give the recommendations.

If you make it configurable it should be *on* by default.

The main use of the deploy-hasingleton is for HA-JMS which needs this behaviour
(restart on merge).
The same will be true of any singleton that is not totally in-memory.

If somebody doesn't want this behaviour, they can always create their
own deploy-hasingleton-ignore-merge with the MBean configured not to restart.

The real solution (which I've advocated for a while) is for this behaviour
to be defined on the MBean so you could do something like:

<mbean ...>
<ha-singleton partition="DefaultPartition" restart-on-merge="true"/>

This is something that can be done in JBoss5 with the decorators (aspects)
and using an annotation rather than specific xml.

<bean ...>
<annotation>@org.jboss.cluster.annotation.HASingleton</annotation>

public @interface HASingleton
{
 String partiionName() default "DefaultPartition";
 boolean restartOnMerge() default true;
}

and even "programmatically":

@HASingleton
public class Queue {}

[adrian.brock]

I'm too lazy to go beyond a StandardMBean approach.

You can specify an alternate interface on a StandardMBean (as long as the MBean
implements that interface).

public interface ExtensionInterface extends MyMBean;

<mbean code="com.acme.My" interface="com.acme.ExtensionInterface"/>

[brian.stansberry]

Thanks for the tip on the alternate interface; that removes one bit of cruft.

If you make it configurable it should be *on* by default.

+1 on this for 4.2 and later, but would you advocate that for a CP? As there's no way of knowing what people have deployed in deploy-hasingleton, I don't think having it change behavior in a CP is good.

[galder.zamarreno]

+1 on this for 4.2 and later

+1

, but would you advocate that for a CP? As there's no way of knowing what people have deployed in deploy-hasingleton, I don't think having it change behavior in a CP is good.

I agree with Brian regarding a possible change of behavior in the CP, off by default in this case.

[brian.stansberry]

BTW, this is all documented at http://wiki.jboss.org/wiki/Wiki.jsp?page=HASingletonAndClusterMerges.

For 4.0.x ended up going the StandardMBean route. Using the 'mbean interface="xxx"' approach would have required changing deploy-hasingleton-service.xml in the CP releases and I didn't want to change a config file in a CP as it could cause problems installing the CP.