Is this example assuming wildfly (active) - wildfly (passive) cluster?
It looks like it is a singleton deployment on application level, and it is specified by the deployment descriptor (singleton-deployment.xml)
I am investigating wildfly (active) - wildfly (active) cluster with Singleton EJB started up in each cluster member, but only one of the Singleton EJB will be active. In this case, I also have concern over a temporary network outage during client invocation on the Singleton EJB and if the cluster will activate another Singleton EJB in this case. If this happens, we will have a split brain issue, and how does wildfly 10 deals with this scenario?
Singleton deployments and services handle network partitions by requiring a quorum. In a N node cluster, using a quorum of N/2 + 1 requires a partition to have a majority membership in order for a singleton election to take place.
This is configured via:
I read your article (https://docs.jboss.org/author/display/WFLY10/Singleton+MSC+services) for Wildfly 10 as to writing a service to manage Singleton EJB so that there will be only one instance active across the cluster.
In the case of the currently active singleton EJB is not accessible due to server shut down, another singleton EJB in the cluster will be activated. What if it is a temporary network issue? Is it possible that another singleton EJB will be activated? If this happens, is it a potential split brain issue?
In addition, apart from writing a service to manage singleton EJB across a cluster, is there any easier way such as configuration file to manage a list of singleton EJB across a cluster?
I would discourage you from mixing EJBs and MSC services as such an approach is susceptible to race conditions - the singleton deployment approach that Rado suggested effectively does the same thing, but in a reliable way.
In the case of the currently active singleton EJB is not accessible due to server shut down, another singleton EJB in the cluster will be activated.
Correct. The shutdown of the server hosting the primary singleton EJB will trigger a cluster membership change that will result in a new primary being elected. Any remote access to this EJB should use appropriate retry logic as there will inevitably be a short period of time where the request to the current primary fails, followed by a short period of time during until the new primary node is elected and started.
What if it is a temporary network issue? Is it possible that another singleton EJB will be activated? If this happens, is it a potential split brain issue?
If the network issue is extremely short, JGroups might see this as a false suspect, in which case no cluster membership changes occur. If the network issue is long enough to trigger a cluster membership change, this will trigger a new singleton election. If the cluster membership change is immediately followed by another membership change (e.g. restoring the original membership), then the a new election will take place after the first one completes. During a singleton election, we always stop the service on the old primary before attempting to start the service on the new primary. If that brief cluster membership change was a network partition (i.e. the node that left didn't actually shutdown, but instead thought that all the other nodes had left), then we rely on the quorum configuration to handle this scenario (i.e. the minority partition will stop any current primary and not elect a new primary).
Thank you very much for the detailed answers!
I understand that there is a subsystem in the HA profile. I was also working on HA singleton on the whole application level, basically turn the JEE application into HA Singleton so that one wildfly instance is active, and the other is on standby mode. In this case, we need to we need to include singleton-deployment.xml in the application. This is basically the cluster with wildfly(active) - wildfly (standby)
However, I am now investigating the cluster of wildfly (active) - wildfly (active) and if I add the singleton-deployment.xml, it will turn the configuration to active-standby.
You also mention that the singleton EJB managed by MSC service is an approach susceptible to race condition. Is there any other way to manage the singleton EJB in a cluster of wildfly (active) - wildfly(active) configuration?
Were you ever able to sort this out? I'm currently using @Singleton EJBs and need to make them singleton across a cluster and don't want to use MSC if that is going to cause issues. The only example I can find is from the Wildfly Quickstarts, which all use MSC services.
Singleton EJB will be started up in each JVM and in an active-active cluster environment, we need to let them to communicate with each other.
One solution is to create an infinispan cache and configure it as replication-cache. This way, any changes will be replicated to each singleton EJB.
Let me know if you run into any issue.