HA election policy

[mohitanchlia]

I am looking for some complete reference to ElectionPolicy in HA Singleton. I am not able to find complete guide. I have gone through Clustering Guide but it's not elaborate enough on HA singleton/ElectionPolicy etc. details.

I have posted this at several places but have no response. I have searched at many places but couldn't find anything

[brian.stansberry]

Following is an extract from a config file used in a test case. It illustrates the usage in 4.x:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE server>
<server>
 <!-- First HASingleton, Election policy is to choose the oldest node as master -->
 <mbean code="org.jboss.ha.singleton.examples.HASingletonMBeanExample"
 name="jboss.examples:service=HASingletonMBeanExample_1">
 </mbean>

 <mbean code="org.jboss.ha.singleton.HASingletonElectionPolicySimple"
 name="jboss.examples:service=HASingletonMBeanExample-HASingletonElectionPolicySimple_1">
 <attribute name="Position">0</attribute>
 </mbean>

 <mbean code="org.jboss.ha.singleton.HASingletonController"
 name="jboss.examples:service=HASingletonMBeanExample-HASingletonController_1">

 <depends optional-attribute-name="ClusterPartition"
 proxy-type="attribute">jboss:service=DefaultPartition</depends>
 <depends>jboss.examples:service=HASingletonMBeanExample_1</depends>
 <depends optional-attribute-name="ElectionPolicy"
 proxy-type="attribute">jboss.examples:service=HASingletonMBeanExample-HASingletonElectionPolicySimple_1</depends>
 <attribute name="TargetName">jboss.examples:service=HASingletonMBeanExample_1</attribute>
 <attribute name="TargetStartMethod">startSingleton</attribute>
 <attribute name="TargetStopMethod">stopSingleton</attribute>
 <attribute name="TargetStopMethodArgument">true</attribute>
 </mbean>

 <!-- Second HASingleton, Election policy is to choose the youngest node as master -->
 <mbean code="org.jboss.ha.singleton.examples.HASingletonMBeanExample"
 name="jboss.examples:service=HASingletonMBeanExample_2">
 </mbean>

 <mbean code="org.jboss.ha.singleton.HASingletonElectionPolicySimple"
 name="jboss.examples:service=HASingletonMBeanExample-HASingletonElectionPolicySimple_2">
 <attribute name="Position">-1</attribute>
 </mbean>

 <mbean code="org.jboss.ha.singleton.HASingletonController"
 name="jboss.examples:service=HASingletonMBeanExample-HASingletonController_2">

 <depends optional-attribute-name="ClusterPartition"
 proxy-type="attribute">jboss:service=DefaultPartition</depends>
 <depends>jboss.examples:service=HASingletonMBeanExample_2</depends>
 <depends optional-attribute-name="ElectionPolicy"
 proxy-type="attribute">jboss.examples:service=HASingletonMBeanExample-HASingletonElectionPolicySimple_2</depends>
 <attribute name="TargetName">jboss.examples:service=HASingletonMBeanExample_2</attribute>
 <attribute name="TargetStartMethod">startSingleton</attribute>
 <attribute name="TargetStopMethod">stopSingleton</attribute>
 <attribute name="TargetStopMethodArgument">true</attribute>
 </mbean>

 <!-- Third HASingleton, Election policy is to choose the 2nd oldest node as master -->
 <mbean code="org.jboss.ha.singleton.examples.HASingletonMBeanExample"
 name="jboss.examples:service=HASingletonMBeanExample_3">
 </mbean>

 <mbean code="org.jboss.ha.singleton.HASingletonElectionPolicySimple"
 name="jboss.examples:service=HASingletonMBeanExample-HASingletonElectionPolicySimple_3">
 <attribute name="Position">1</attribute>
 </mbean>

 <mbean code="org.jboss.ha.singleton.HASingletonController"
 name="jboss.examples:service=HASingletonMBeanExample-HASingletonController_3">

 <depends optional-attribute-name="ClusterPartition"
 proxy-type="attribute">jboss:service=DefaultPartition</depends>
 <depends>jboss.examples:service=HASingletonMBeanExample_3</depends>
 <depends optional-attribute-name="ElectionPolicy"
 proxy-type="attribute">jboss.examples:service=HASingletonMBeanExample-HASingletonElectionPolicySimple_3</depends>
 <attribute name="TargetName">jboss.examples:service=HASingletonMBeanExample_3</attribute>
 <attribute name="TargetStartMethod">startSingleton</attribute>
 <attribute name="TargetStopMethod">stopSingleton</attribute>
 <attribute name="TargetStopMethodArgument">true</attribute>
 </mbean>

 <!-- Fourth HASingleton, No election policy defined. By default, the oldest node is selected -->
 <mbean code="org.jboss.ha.singleton.examples.HASingletonMBeanExample"
 name="jboss.examples:service=HASingletonMBeanExample_4">
 </mbean>

 <mbean code="org.jboss.ha.singleton.HASingletonController"
 name="jboss.examples:service=HASingletonMBeanExample-HASingletonController_4">

 <depends optional-attribute-name="ClusterPartition"
 proxy-type="attribute">jboss:service=DefaultPartition</depends>
 <depends>jboss.examples:service=HASingletonMBeanExample_4</depends>
 <attribute name="TargetName">jboss.examples:service=HASingletonMBeanExample_4</attribute>
 <attribute name="TargetStartMethod">startSingleton</attribute>
 <attribute name="TargetStopMethod">stopSingleton</attribute>
 <attribute name="TargetStopMethodArgument">true</attribute>
 </mbean>
</server>

There are four sets of beans above. Basically, in each set there is a mbean service being controlled, an mbean that implements the HASingletonElectionPolicy interfacr to conduct the election as to who is master, and an HASingletonController that calls on the election policy and starts/stops the service being controlled.

In 4.x we ship a single implementation of HASingletonElectionPolicy, org.jboss.ha.singleton.HASingletonElectionPolicySimple. This implementation takes a single configuration element "Position", which specifies the ordinal position in the current cluster topology of the node that should be master. 0 == first, -1 == last, 1 == second, -2 == second from last, etc. Cluster topology means, for example, you have nodes A, B, C that have a service deployed, and they started in that order, so the cluster topology for the service would be {A, B, C}. So, position 0 would be A, position -1 would be C, position 1 or -2 would be B. The implementation uses a modulus, so position 3 would be A, 4 would be B etc.

If no policy is specified, as in the last set of beans above, the default is to use HASingletonElectionPolicySimple with a position of 0.

You can create your own implementation of HASingletonElectionPolicy and we provide a base class to start from org.jboss.ha.singleton.HASingletonElectionPolicyBase.

Note that in AS 5, the HASingletonElectionPolicy interface is changed (simplified/rationalized), with the HASingletonElectionPolicyBase class eliminated. HASingletonElectionPolicySimple still works as described above. The AS also ships with a new policy impl, PreferredMasterElectionPolicy. This is an extension of HASingletonElectionPolicySimple that adds a new configuration property "preferredMaster" that lets you specify that you want a particular node to be the master if it is available. Following illustrates its use. (Note that here the config is using the JBoss Microcontainer config format rather than the legacy -service.xml format. The legacy format still works, though):

 <!-- 5th HASingleton, PreferredMaster set to 192.168.0.10:1099. If not available, fallback election policy is to choose the oldest node as master -->
 <bean class="org.jboss.ha.singleton.examples.HASingletonMBeanExample"
 name="HASingletonMBeanExample_5">
 <annotation>@org.jboss.aop.microcontainer.aspects.jmx.JMX(name="jboss.examples:service=HASingletonMBeanExample_5", exposedInterface=org.jboss.ha.singleton.examples.HASingletonMBeanExampleMBean.class, registerDirectly=true)</annotation>
 </bean>

 <bean class="org.jboss.ha.singleton.PreferredMasterElectionPolicy"
 name="PreferredMasterElectionPolicy_1">
 <property name="position">0</property>
 <property name="preferredMaster">192.168.0.10:1099</property>
 </bean>

 <bean class="org.jboss.ha.singleton.HASingletonController"
 name="HASingletonController_5">

 <annotation>@org.jboss.aop.microcontainer.aspects.jmx.JMX(name="test:service=HASingletonController_5", exposedInterface=org.jboss.ha.singleton.HASingletonControllerMBean.class, registerDirectly=true)</annotation>

 <property name="clusterPartition"><inject bean="HAPartition"/></property>
 <property name="electionPolicy"><inject bean="PreferredMasterElectionPolicy_1"/></property>
 <property name="target"><inject bean="HASingletonMBeanExample_5"/></property>
 <property name="targetStartMethod">startSingleton</property>
 <property name="targetStopMethod">stopSingleton</property>
 <property name="targetStopMethodArgument">true</property>
 </bean>

[mohitanchlia]

Thanks for the explanation - it's easy to understand. I have some questions though:

1. Take a scenario where there are 4 nodes A,B,C,D but only 2 nodes have come up so far. What happens to HASingletonController_3 if only first 2 nodes are running. If I understand correctly it will do a mod of (3 % 2). Which means that HASingletonController_3 will be started on node B. Now what happens if Node 3 comes up and eventually Node 4. Would the master change everytime a Node joins a cluster?

2. In scenario where all nodes are running, is it correct to say that HASingletonController will be in "start" state in only one of the nodes and all other nodes it will be either created/stopped.

3. Why does HASingletonController depends on HASingletonMBeanExample_1, shouldn't it be other way round? I thought HASingletonController is responsible for starting/stopping the MBean.

[brian.stansberry]

"mohitanchlia" wrote:

1. Take a scenario where there are 4 nodes A,B,C,D but only 2 nodes have come up so far. What happens to HASingletonController_3 if only first 2 nodes are running. If I understand correctly it will do a mod of (3 % 2). Which means that HASingletonController_3 will be started on node B. Now what happens if Node 3 comes up and eventually Node 4. Would the master change everytime a Node joins a cluster?

Yes. This is clearly a disadvantage to using a negative number for the position.

With your A, B, C, D example, if you had a value of 3, the master would change 4 times as well, but then would stabilize if you added more nodes.

Basically, any value other than 0 would be more prone to having the master change than 0 is. 0 only changes if the first node running stops.

2. In scenario where all nodes are running, is it correct to say that HASingletonController will be in "start" state in only one of the nodes and all other nodes it will be either created/stopped.

No, the HASingletonController comes into start state on all nodes during normal startup. It's a special service that manages a target mbean, e.g. HASingletonMBeanExample_1. It itself is not meant to be a singleton.

3. Why does HASingletonController depends on HASingletonMBeanExample_1, shouldn't it be other way round? I thought HASingletonController is responsible for starting/stopping the MBean.

HASingletonMBeanExample_1 is brought to the start state when it's deployed, just like any other mbean. What a service that's controlled by an HASingletonController does is conceptually add a couple new lifecycle stagess that come between "started" and "stopped". Call them "singletonStarted" and "singletonStopped". The service exposes 2 no-arg or single-arg methods that the HASingletonController can call to bring the service to the "singletonStarted" and "singletonStopped" states. Those methods can be named whatever you like; you tell the HASingletonController their names (and optionally a single param):

 <attribute name="TargetStartMethod">startSingleton</attribute>
 <attribute name="TargetStopMethod">stopSingleton</attribute>
 <attribute name="TargetStopMethodArgument">true</attribute>

See http://www.jboss.org/file-access/default/members/jbossas/freezone/docs/Clustering_Guide/beta422/html/ch05s11s02.html[/url]

[mohitanchlia]

Thanks. so if we use below config then Barrier will be in start state in only in one node?

<mbean code="org.jboss.system.BarrierController" name="efe.ha:service=HASingletonController_1,type=BarrierController">
 <!-- Dependent services will depend on this mbean -->
 <attribute name="BarrierObjectName">efe.ha:service=HASingletonController_1,type=Barrier</attribute>
 <!-- Whether to subscribe for notifications after startup -->
 <attribute name="DynamicSubscriptions">false</attribute> ....

Also, when Barrier starts does it deliver some kind of notification to the service or it just starts? As I understand Barrier is used for those services that don't follow services lifecycle like start/stop. So how does Barrier is able to stop or start those services like EJB, MDB?

[mohitanchlia]

Could someone help me understand my earlier post regarding Barrier

[brian.stansberry]

The config you posted is partial, so I don't know if it is right or wrong.

In the AS 4.x distributions see the server/all/deploy/deploy-hasingleton-service.xml for an example of how to deploy a BarrierController mbean in combination with an HASingletonController mbean. With such a config, the Barrier will only be in the start state on one node.

[mohitanchlia]

Thanks I already looked at it. How is Barrier able to start or stop EJBs, MDBs that are not mbean and don't follow service life cycle i.e start/stop/create etc.

Also, what happens if Election policy like pickSingleton has a bug and chooses 2 nodes as Master at the same time?

[brian.stansberry]

"mohitanchlia" wrote:
Thanks I already looked at it. How is Barrier able to start or stop EJBs, MDBs that are not mbean and don't follow service life cycle i.e start/stop/create etc.

In JBoss 4.x? Everthing ends up as an mbean. You add a a "depends" element to the relevant deployment descriptor (jboss-app.xml if an ear, jboss.xml for for EJB/MDB).

Also, what happens if Election policy like pickSingleton has a bug and chooses 2 nodes as Master at the same time?

Then whatever depends on the barrier ends up being started on 2 nodes at the same time.

[mohitanchlia]

Thanks. I got few more questions:

- Does HAPartition notify HASingletonController when node goes down and then HASingletonController that leads in invocation of pickSingleton.

- I meant to ask how is Barrier able to start/stop EJB. As I understand EJBs don't follow Jboss service life cycle. How is it internally able to start/stop EJBs.

- Also, when is pickSingleton Called.

Thanks for answering my questions.