11 Replies Latest reply on Oct 6, 2014 2:43 PM by weboctave

    Multiple node elected as Singleton service provider

    bsudhananthan

      I'm using jboss-eap-6.1.0. standalone-full-ha configuration.

      Singleton service start method has been triggered in multiple nodes i.e both master and slave nodes has been elected as service provider for "MySingletonService.SINGLETON_SERVICE_NAME".

       

      My singleton service initiator ClusterServiceInitiater.java looks like :

       

      @Singleton

      @Startup

      public class ClusterServiceInitiater

      {

          /**

           * create the service during startup

           */

          @PostConstruct

          protected void clusterSingletonStartup()

          {

              MySingletonService mySingletonService = new MySingletonService();

              SingletonService<String> singletonService = new SingletonService<String>(mySingletonService, MySingletonService.SINGLETON_SERVICE_NAME);

              CurrentServiceContainer.getServiceContainer().dumpServices();

              ServiceController<String> controller =  singletonService.build(CurrentServiceContainer.getServiceContainer())

                      .addDependency(ServiceName.JBOSS.append("cluster", "singleton"),

                              GroupMembershipNotifier.class, mySingletonService.getEnvInjector()).install();

              //controller.setMode(ServiceController.Mode.ACTIVE);

          }

       

          /**

           * Remove the service during undeploy or shutdown

           */

          @PreDestroy

          protected void destroy() {

            ServiceController<?> controller = CurrentServiceContainer.getServiceContainer().getRequiredService(MySingletonService.SINGLETON_SERVICE_NAME);

            controller.setMode(ServiceController.Mode.REMOVE);

          }

       

       

       

       

      }

       

      And the singleton class MYSingletonService.java looks like:

      public class MySingletonService implements Service<String>, GroupMembershipListener

      {

          private static Log logger = LogFactory.getLog(MySingletonService.class);

          public static final ServiceName SINGLETON_SERVICE_NAME = ServiceName.JBOSS.append("MY_Singleton_Service");

          private final InjectedValue<GroupMembershipNotifier> notifier = new InjectedValue<GroupMembershipNotifier>();

       

          @Override

          public String getValue() throws IllegalStateException,

                  IllegalArgumentException {

              // TODO Auto-generated method stub

              return null;

          }

       

          @Override

          public void start(StartContext arg0) throws StartException {

              logger.info("::::Inside Start Of Singleton:::");

          }

       

          @Override

          public void stop(StopContext arg0) {

              logger.info("::::Inside Start Of Singleton:::");

          }

       

          @Override

          public void membershipChanged(List<ClusterNode> arg0,

                  List<ClusterNode> arg1, List<ClusterNode> arg2) {

              // TODO Auto-generated method stub

       

          }

       

          @Override

          public void membershipChangedDuringMerge(List<ClusterNode> arg0,

                  List<ClusterNode> arg1, List<ClusterNode> arg2,

                  List<List<ClusterNode>> arg3) {

              // TODO Auto-generated method stub

       

          }

       

          public Injector<GroupMembershipNotifier> getEnvInjector() {

              return this.notifier;

          }

      }

       

       

       

      What happens is when i start two nodes at same time, Singleton service's start() method has been triggered in multiple nodes i.e both master and slave nodes has been elected as service provider of "MySingletonService.SINGLETON_SERVICE_NAME". Can any on explain me why this is happening.

       

      Thanks in Advance,

      Sudhananthan B.

        • 1. Re: Multiple node elected as Singleton service provider
          wdfink

          You should use the Serviceactivator pattern to start, see jboss-eap-quickstarts/cluster-ha-singleton.

           

          Also did you see that the cluster is build correct and the nodes see each other as one cluster?

          • 2. Re: Multiple node elected as Singleton service provider
            rhusar

            The SingletonService itself is going to start on all nodes, it is the wrapped service that is going to be started only on one node.

            • 3. Re: Multiple node elected as Singleton service provider
              bsudhananthan

              Thanks

              Wolf-Dieter Fink wrote:

               

              You should use the Serviceactivator pattern to start, see jboss-eap-quickstarts/cluster-ha-singleton.

               

              Also did you see that the cluster is build correct and the nodes see each other as one cluster?

               

              I used Serviceactivator to start the singleton service and i got the same result.

               

              My analysis related to this issue

              The two nodes are : [sudha_1/singleton, ram_2/singleton]

              In org.jboss.as.clustering.service.ServiceProviderRegistryService i've inserted some log infos as shown below

               

              @Override

                  public void register(final String service, Listener listener) {

                        this.listeners.put(service, listener);

                       final ClusterNode node = this.notifier.getClusterNode();

                       LOGGER.info("Node :"+node);

                       LOGGER.info("Existing Cluster Nodes :"+this.notifier.getClusterNodes());

                      Operation<Set<ClusterNode>> operation = new Operation<Set<ClusterNode>>() {

                          @Override

                          public Set<ClusterNode> invoke(Cache<String, Map<ClusterNode, Void>> cache) {

                              Map<ClusterNode, Void> map = cache.putIfAbsent(service, null);

                              if (!map.containsKey(node)) {

                                  map.put(node, null);

                              }

                              return map.keySet();

                          }

                      };

                      Set<ClusterNode> nodes = this.invoker.invoke(this.cache, operation);

                     LOGGER.info("Node Elected from Cache :"+nodes);

                      listener.serviceProvidersChanged(nodes, false);

                  }

                    Note : I've added above log(Strikethrough in the above quoted register method) in the org.jboss.as.clustering.service.ServiceProviderRegistryService class and the outcome were given below

               

              In Master Node [sudha_1] :

               

                      Node : sudha_1/singleton

                      Existing Cluster Nodes : [sudha_1/singleton, ram_2/singleton]

                      Node Elected from Cache : [sudha_1/singleton, ram_2/singleton]

               

              In Slave Node [ram_2]  :

               

                      Node : ram_2/singleton

                      Existing Cluster Nodes : [sudha_1/singleton, ram_2/singleton]

                     Node Elected from Cache : [ram_2/singleton]

               

              So what I've analysed is while getting through the ServiceProviderRegistryService.register() method from slave node Node Elected from Cache : [ram_2/singleton]  which differ from Master Node result, but it has the knowledge about the master node(i.e @Slave Node : Existing Cluster Nodes : [sudha_1/singleton, ram_2/singleton]).

               

              I think it might be helpful.

               

              Thanks,

              Sudhananthan B.

              • 4. Re: Multiple node elected as Singleton service provider
                wdfink

                The Service<> start method should only be called if the node is elected as master.

                 

                Did you check that the clusteris build correct?

                • 5. Re: Multiple node elected as Singleton service provider
                  bsudhananthan

                  The Service<> start method should only be called if the node is elected as master.

                  Did you check that the clusteris build correct?

                  Yes I've checked it. And for your kind notice, the issue only reproduced when both server starts at some micro second interval difference. And that too is indeterminable, since it is not occurs frequently.

                   

                  I've specified the result of my analysis in the previous reply under My analysis related to this issue section.

                  • 6. Re: Multiple node elected as Singleton service provider
                    wdfink

                    Is one service stopped directly or are they continue running both? In that case I would think this is a bug.

                    Is that reproducable with EAP6.2 and WildFly (upstream)?

                    1 of 1 people found this helpful
                    • 7. Re: Multiple node elected as Singleton service provider
                      bsudhananthan

                      Wolf-Dieter Fink wrote:

                       

                      Is one service stopped directly or are they continue running both?

                      Yes, the service gets stopped when other one gets elected as singleton service provider. Is there any other way to restrict the service getting elected while the other one was already elected and serving.

                      • 8. Re: Multiple node elected as Singleton service provider
                        wdfink

                        In that case it sounds like a rece-condition during startup to me. The only way is to start the second instance with delay.

                        Could you reproduce it with EAP6.2 or WildFly? If yes please file a JIRA and attach the logfiles, best with trace for org.jgoups and org.jboss.as.clustering

                        • 9. Re: Multiple node elected as Singleton service provider
                          weboctave

                          This happened in EAP 6.2.

                          • 10. Re: Multiple node elected as Singleton service provider
                            wdfink

                            Alan,

                             

                            is it exact the same issue? You should provide more details or maybe open a new thread

                            • 11. Re: Multiple node elected as Singleton service provider
                              weboctave

                              I believed so.  I grep from server.log from our servers.  We have 6 servers (server1-server6) in the same server group.  2 servers on each host (host1 - host3).  Domain controller is on host1.

                              At 03:00, both host1:atp-server1 and host3:atp-server5 are elected as singleton provider.    The servers were restarted s

                               

                              atpappprod1/atp-server1/log/server.log:10/04/14 02:54:26,467 INFO  [org.jboss.as.clustering.singleton] (notification-thread-0) JBAS010342: host1:atp-server1/singleton elected as the singleton provider of the jboss.atp.singleton.timer service

                              atpappprod1/atp-server1/log/server.log:10/04/14 02:54:26,565 INFO  [org.jboss.as.clustering.singleton] (notification-thread-0) JBAS010342: host1:atp-server1/singleton elected as the singleton provider of the jboss.atp.singleton.timer service

                              atpappprod1/atp-server1/log/server.log:10/04/14 02:54:26,691 INFO  [org.jboss.as.clustering.singleton] (notification-thread-0) JBAS010342: host1:atp-server1/singleton elected as the singleton provider of the jboss.atp.singleton.timer service

                              atpappprod1/atp-server1/log/server.log:10/04/14 02:58:04,596 INFO  [org.jboss.as.clustering.singleton] (ServerService Thread Pool -- 62) JBAS010342: host1:atp-server1/singleton elected as the singleton provider of the jboss.atp.singleton.timer service

                              atpappprod1/atp-server1/log/server.log:10/04/14 02:58:31,353 INFO  [org.jboss.as.clustering.singleton] (notification-thread-0) JBAS010342: host1:atp-server1/singleton elected as the singleton provider of the jboss.atp.singleton.timer service

                              atpappprod1/atp-server1/log/server.log:10/04/14 02:59:49,554 INFO  [org.jboss.as.clustering.singleton] (notification-thread-0) JBAS010342: host1:atp-server1/singleton elected as the singleton provider of the jboss.atp.singleton.timer service

                              atpappprod1/atp-server1/log/server.log:10/04/14 02:59:49,769 INFO  [org.jboss.as.clustering.singleton] (notification-thread-0) JBAS010342: host1:atp-server1/singleton elected as the singleton provider of the jboss.atp.singleton.timer service

                              atpappprod1/atp-server1/log/server.log:10/04/14 03:00:02,787 INFO  [org.jboss.as.clustering.singleton] (notification-thread-0) JBAS010342: host1:atp-server1/singleton elected as the singleton provider of the jboss.atp.singleton.timer service

                              atpappprod1/atp-server1/log/server.log:10/04/14 03:00:28,931 INFO  [org.jboss.as.clustering.singleton] (notification-thread-0) JBAS010342: host1:atp-server1/singleton elected as the singleton provider of the jboss.atp.singleton.timer service

                              atpappprod2/atp-server3/log/server.log:10/04/14 02:54:26,534 INFO  [org.jboss.as.clustering.singleton] (notification-thread-0) JBAS010342: host1:atp-server1/singleton elected as the singleton provider of the jboss.atp.singleton.timer service

                              atpappprod2/atp-server3/log/server.log:10/04/14 02:59:49,618 INFO  [org.jboss.as.clustering.singleton] (ServerService Thread Pool -- 76) JBAS010342: host1:atp-server1/singleton elected as the singleton provider of the jboss.atp.singleton.timer service

                              atpappprod2/atp-server3/log/server.log:10/04/14 02:59:49,768 INFO  [org.jboss.as.clustering.singleton] (notification-thread-0) JBAS010342: host1:atp-server1/singleton elected as the singleton provider of the jboss.atp.singleton.timer service

                              atpappprod2/atp-server3/log/server.log:10/04/14 03:00:02,787 INFO  [org.jboss.as.clustering.singleton] (notification-thread-0) JBAS010342: host1:atp-server1/singleton elected as the singleton provider of the jboss.atp.singleton.timer service

                              atpappprod2/atp-server3/log/server.log:10/04/14 03:00:18,792 INFO  [org.jboss.as.clustering.singleton] (notification-thread-0) JBAS010342: host1:atp-server1/singleton elected as the singleton provider of the jboss.atp.singleton.timer service

                              atpappprod3/atp-server5/log/server.log:10/04/14 02:59:49,777 INFO  [org.jboss.as.clustering.singleton] (ServerService Thread Pool -- 68) JBAS010342: host3:atp-server5/singleton elected as the singleton provider of the jboss.atp.singleton.timer service

                              atpappprod3/atp-server5/log/server.log:10/04/14 03:00:02,787 INFO  [org.jboss.as.clustering.singleton] (notification-thread-0) JBAS010342: host3:atp-server5/singleton elected as the singleton provider of the jboss.atp.singleton.timer service

                              atpappprod3/atp-server5/log/server.log:10/04/14 03:00:18,808 INFO  [org.jboss.as.clustering.singleton] (notification-thread-0) JBAS010342: host3:atp-server5/singleton elected as the singleton provider of the jboss.atp.singleton.timer service

                              atpappprod3/atp-server6/log/server.log:10/04/14 02:54:26,515 INFO  [org.jboss.as.clustering.singleton] (notification-thread-0) JBAS010342: host1:atp-server1/singleton elected as the singleton provider of the jboss.atp.singleton.timer service

                              atpappprod3/atp-server6/log/server.log:10/04/14 02:54:26,608 INFO  [org.jboss.as.clustering.singleton] (notification-thread-0) JBAS010342: host1:atp-server1/singleton elected as the singleton provider of the jboss.atp.singleton.timer service

                              atpappprod3/atp-server6/log/server.log:10/04/14 03:00:18,819 INFO  [org.jboss.as.clustering.singleton] (ServerService Thread Pool -- 65) JBAS010342: host3:atp-server5/singleton elected as the singleton provider of the jboss.atp.singleton.timer service