8 Replies Latest reply on Apr 4, 2007 3:01 PM by Brian Stansberry

    HASingleton and partition merges

    Brian Stansberry Master

      Discussion related to http://jira.jboss.com/jira/browse/JBAS-4229.

      The way a clustered service is aware that a 'split brain' situation has occurred and is now healed is by receiving a MergeView from JGroups. Problem here is that HASingletonSupport does not receive view change notifications from JGroups -- it receives service topology change notifications from DistributedReplicantManager. (The topology of a service can change for reasons other than view changes; e.g. undeploy of the service on a node.) The topology change notification does not include any information as to whether the change was due to a merge.

      In 4.x this can be somewhat shakily resolved by having the topology change callback do some extra work to try to figure out from HAPartition whether a MergeView has occurred. That is, register with the HAPartition for membership change events and track whether they are due to merges.

      I say shakily resolved because the topology change notifications and the view change notifications come on separate threads with no guaranteed ordering. If numerous events are received in short order, the potential to have mis-correlation between topology changes and view changes certainly exists.

      In 5.x a cleaner solution is possible, since in a major version I can change the DRM.replicantListener interface. I can add a boolean flag to the callback so the DRM can directly notify the singleton that the change was due to a merge.

        • 1. Re: HASingleton and partition merges
          Brian Stansberry Master

          The problem of ordering notifications can likely be solved.

          Another issue is whether this kind of restart-on-merge behavior should be configurable or even turned on by default. HA-JMS needs this; for other use cases having the singleton restart may be undesirable. This is particularly an issue with the HASingletonDeployer mbean, since a restart there means redeploying everything in deploy-hasingleton, a potentially lengthy process.

          I've no problem with adding a config flag for this, although that complicates backporting it for CP releases.

          • 2. Re: HASingleton and partition merges
            Galder Zamarreño Master

            I think it needs configuring too but I would leave it as off by default so that it mantains previous 4.0.x behaivour and then customers can set it to true for recommended services, such as HA-JMS.

            What complications can you see in backporting this? The patch instructions could explain the flag and give the recommendations.

            • 3. Re: HASingleton and partition merges
              Brian Stansberry Master

              The complication with backporting is:

              1) New config flag == change to mbean interface
              2) Change to interface is illegal in a CP, hence have to create a new interface and new probably a trivial new class that exposes it (I'm too lazy to go beyond a StandardMBean approach.
              3) Don't want to change deploy-hasingleton-service.xml to use this new class/interface, as that's too big a change for a CP (changing config file could break people who've already altered it.)

              All this is resolvable, it's just extra hassle. And the result is that a fix will be in the codebase, but to take advantage of it users will have to make manual configuration changes.


              (BTW, users can't set the flag to true for HA-JMS per se. They can set it to true for the jboss.ha:service=HASingletonDeployer service, which in essence means setting it to true for the entire deploy-hasingleton directory. If they wanted the behavior only for HA-JMS and had other deploy-hasingleton content, they'd have to separate the content into another dir and create a parallel HASingletonController to deploy that dir.)

              • 4. Re: HASingleton and partition merges
                Adrian Brock Master

                 

                "galder.zamarreno@jboss.com" wrote:
                I think it needs configuring too but I would leave it as off by default so that it mantains previous 4.0.x behaivour and then customers can set it to true for recommended services, such as HA-JMS.

                What complications can you see in backporting this? The patch instructions could explain the flag and give the recommendations.


                If you make it configurable it should be *on* by default.

                The main use of the deploy-hasingleton is for HA-JMS which needs this behaviour
                (restart on merge).
                The same will be true of any singleton that is not totally in-memory.

                If somebody doesn't want this behaviour, they can always create their
                own deploy-hasingleton-ignore-merge with the MBean configured not to restart.

                The real solution (which I've advocated for a while) is for this behaviour
                to be defined on the MBean so you could do something like:
                <mbean ...>
                <ha-singleton partition="DefaultPartition" restart-on-merge="true"/>
                


                This is something that can be done in JBoss5 with the decorators (aspects)
                and using an annotation rather than specific xml.

                <bean ...>
                <annotation>@org.jboss.cluster.annotation.HASingleton</annotation>
                
                public @interface HASingleton
                {
                 String partiionName() default "DefaultPartition";
                 boolean restartOnMerge() default true;
                }
                
                and even "programmatically":
                
                @HASingleton
                public class Queue {}
                



                • 5. Re: HASingleton and partition merges
                  Adrian Brock Master

                   

                  "bstansberry@jboss.com" wrote:
                  I'm too lazy to go beyond a StandardMBean approach.


                  You can specify an alternate interface on a StandardMBean (as long as the MBean
                  implements that interface).

                  public interface ExtensionInterface extends MyMBean;
                  
                  <mbean code="com.acme.My" interface="com.acme.ExtensionInterface"/>


                  • 6. Re: HASingleton and partition merges
                    Brian Stansberry Master

                    Thanks for the tip on the alternate interface; that removes one bit of cruft.

                    "adrian@jboss.org" wrote:

                    If you make it configurable it should be *on* by default.


                    +1 on this for 4.2 and later, but would you advocate that for a CP? As there's no way of knowing what people have deployed in deploy-hasingleton, I don't think having it change behavior in a CP is good.

                    • 7. Re: HASingleton and partition merges
                      Galder Zamarreño Master

                       

                      "bstansberry@jboss.com" wrote:
                      +1 on this for 4.2 and later


                      +1

                      "bstansberry@jboss.com" wrote:
                      , but would you advocate that for a CP? As there's no way of knowing what people have deployed in deploy-hasingleton, I don't think having it change behavior in a CP is good.


                      I agree with Brian regarding a possible change of behavior in the CP, off by default in this case.

                      • 8. Re: HASingleton and partition merges
                        Brian Stansberry Master

                        BTW, this is all documented at http://wiki.jboss.org/wiki/Wiki.jsp?page=HASingletonAndClusterMerges.

                        For 4.0.x ended up going the StandardMBean route. Using the 'mbean interface="xxx"' approach would have required changing deploy-hasingleton-service.xml in the CP releases and I didn't want to change a config file in a CP as it could cause problems installing the CP.