5 Replies Latest reply on May 15, 2014 3:49 PM by wdfink

    Issue with HA Singleton Service

    sreeja_e_j

      We have an application in which the singleton node is decided based on the implementation (Create a singleton service) given in the link below

      https://access.redhat.com/site/documentation/en-US/JBoss_Enterprise_Application_Platform/6/html/Development_Guide/Implement_an_HA_Singleton.html

      If the node is the singleton provider, certain threads are executed, else they are not.  We have a situation where in multiple nodes are executing this threads.

       

      Could this be because of any configuration issues ? Are there any known issues with "HA Singleton Service". We are using JBoss 7.1.1.

        • 1. Re: Issue with HA Singleton Service
          sreeja_e_j

          This turned out to be a network issue between the master and the slave node.

           

          When the split occurred both the Master and Slave started the singleton service (Split-Brain Issue) .

          However, when the Merge occurred, the "Service Stop" request  is not being received on none of the Nodes.

          As a result,  the singleton service continues to run on both the nodes after merge.

           

          We do see an issue called out in https://issues.jboss.org/browse/AS7-5218


          But need to know if this can be handled in JBOSS 7.1.1

          • 2. Re: Issue with HA Singleton Service
            wdfink

            If you look to the JIRA the issue is resolved in 7.1.3 or EAP6.1.0.Alpha.

            AS7.1.1 will behave the same as 7.1.0.

             

            You need to upgrade to a newer version. EAP6.1.0.Alpha is available for free. AS7.1.3 or 7.2 can be build from the sources at github.

            Also you can download WildFly 8.0.0.Final

            • 3. Re: Issue with HA Singleton Service
              sreeja_e_j

              Thanks for the reply.

               

              Unfortunately, we wont be able to upgrade to 7.1.3. We are looking at alternate options.

               

              We are trying to implement GroupMembershipListener and wait for  membershipChangedDuringMerge() to be invoked..

              We are observing that this method is being invoked during a Merge ( after a SplitBrain situation )

               

              Can we force a re-election at this point ? Is there a way to achieve this ?

              • 4. Re: Issue with HA Singleton Service
                sreeja_e_j

                We have implemented the GroupMembershipListener and in the


                public void membershipChangedDuringMerge(List<ClusterNode> arg0,

                  List<ClusterNode> arg1, List<ClusterNode> arg2,

                  List<List<ClusterNode>> arg3) {

                {

                     <if the first node in the viewList is not the same as the nodename >

                     {

                          ServiceController<?> controller = CurrentServiceContainer.getServiceContainer().getRequiredService(<service name>);

                          SingletonService<String> ss = (SingletonService<String>) controller.getService();

                         Set set = new HashSet(arg2);

                         ss.serviceProvidersChanged(set, true);

                     }

                }

                This seems to be helping our error scenario - 2 Nodes Running as Singleton Provider on Merge

                 

                Do let us know if you have any comments on this

                • 5. Re: Issue with HA Singleton Service
                  wdfink

                  If you test your workaround and it works for you you might stay with it.

                   

                  But I would prefer to upgrade or integrate the fix in the version you use as I know that such workarounds live a long time as noone remember why it was installed