8 Replies Latest reply on May 7, 2019 11:41 PM by wwang2016

    Singleton EJB in a cluster

    wwang2016

      Hi

       

      There should be one active Singleton EJB across a cluster.

      what if the EJB is not accessible for a network reason for a short while? will another instance be active?

       

      Thanks,

       

      Wayne

        • 1. Re: Singleton EJB in a cluster
          rhusar

          Take a look at this quickstart quickstart/ha-singleton-deployment at 11.x · wildfly/quickstart · GitHub  which demonstrates how to deploy a singleton EJB the recommended way.

          • 2. Re: Singleton EJB in a cluster
            wwang2016

            Hi Radoslav,

             

            Is this example assuming wildfly (active) - wildfly (passive) cluster?

             

            It looks like it is a singleton deployment on application level, and it is specified by the deployment descriptor (singleton-deployment.xml)

             

            I am investigating wildfly (active) - wildfly (active) cluster with Singleton EJB started up in each cluster member, but only one of the Singleton EJB will be active. In this case, I also have concern over a temporary network outage during client invocation on the Singleton EJB and if the cluster will activate another Singleton EJB in this case. If this happens, we will have a split brain issue, and how does wildfly 10 deals with this scenario?

             

            Thanks,

             

            Wayne

            • 3. Re: Singleton EJB in a cluster
              pferraro

              Singleton deployments and services handle network partitions by requiring a quorum.  In a N node cluster, using a quorum of N/2 + 1 requires a partition to have a majority membership in order for a singleton election to take place.

              This is configured via:

              /subsystem=singleton/singleton-policy=default:write-attribute(name=quorum, value=...)
              
              • 4. Re: Singleton EJB in a cluster
                wwang2016

                Hi Paul,

                 

                I read your article (https://docs.jboss.org/author/display/WFLY10/Singleton+MSC+services) for Wildfly 10 as to writing a service to manage Singleton EJB so that there will be only one instance active across the cluster.

                 

                In the case of the currently active singleton EJB is not accessible due to server shut down, another singleton EJB in the cluster will be activated. What if it is a temporary network issue? Is it possible that another singleton EJB will be activated? If this happens, is it a potential split brain issue?

                 

                In addition, apart from writing a service to manage singleton EJB across a cluster, is there any easier way such as configuration file to manage a list of singleton EJB across a cluster?

                 

                Thanks,

                 

                Wayne

                • 5. Re: Singleton EJB in a cluster
                  pferraro

                  I would discourage you from mixing EJBs and MSC services as such an approach is susceptible to race conditions - the singleton deployment approach that Rado suggested effectively does the same thing, but in a reliable way.

                  In the case of the currently active singleton EJB is not accessible due to server shut down, another singleton EJB in the cluster will be activated.

                  Correct.  The shutdown of the server hosting the primary singleton EJB will trigger a cluster membership change that will result in a new primary being elected.  Any remote access to this EJB should use appropriate retry logic as there will inevitably be a short period of time where the request to the current primary fails, followed by a short period of time during until the new primary node is elected and started.

                  What if it is a temporary network issue? Is it possible that another singleton EJB will be activated? If this happens, is it a potential split brain issue?

                  If the network issue is extremely short, JGroups might see this as a false suspect, in which case no cluster membership changes occur.  If the network issue is long enough to trigger a cluster membership change, this will trigger a new singleton election.  If the cluster membership change is immediately followed by another membership change (e.g. restoring the original membership), then the a new election will take place after the first one completes.  During a singleton election, we always stop the service on the old primary before attempting to start the service on the new primary.  If that brief cluster membership change was a network partition (i.e. the node that left didn't actually shutdown, but instead thought that all the other nodes had left), then we rely on the quorum configuration to handle this scenario (i.e. the minority partition will stop any current primary and not elect a new primary).

                  • 6. Re: Singleton EJB in a cluster
                    wwang2016

                    Hi Paul,

                     

                    Thank you very much for the detailed answers!

                     

                    I understand that there is a subsystem in the HA profile. I was also working on HA singleton on the whole application level, basically turn the JEE application into HA Singleton so that one wildfly instance is active, and the other is on standby mode. In this case, we need to we need to include singleton-deployment.xml in the application. This is basically the cluster with wildfly(active) - wildfly (standby)

                     

                    However, I am now investigating the cluster of wildfly (active) - wildfly (active) and if I add the singleton-deployment.xml, it will turn the configuration to active-standby.

                     

                    You also mention that the singleton EJB managed by MSC service is an approach susceptible to race condition. Is there any other way to manage the singleton EJB in a cluster of wildfly (active) - wildfly(active) configuration?

                     

                    Thanks

                     

                    Wayne

                    • 7. Re: Singleton EJB in a cluster
                      jputney

                      Were you ever able to sort this out? I'm currently using @Singleton EJBs and need to make them singleton across a cluster and don't want to use MSC if that is going to cause issues. The only example I can find is from the Wildfly Quickstarts, which all use MSC services.

                      • 8. Re: Singleton EJB in a cluster
                        wwang2016

                        Singleton EJB will be started up in each JVM and in an active-active cluster environment, we need to let them to communicate with each other.

                         

                        One solution is to create an infinispan cache and configure it as replication-cache. This way, any changes will be replicated to each singleton EJB.

                         

                        Let me know if you run into any issue.

                         

                        Good luck

                         

                        Wayne