I created a singleton service based on the quickstart example (). For a single cluster group, things work perfectly. i.e., only one cluster node runs the service and when the same goes down, another cluster node takes over. We observe that in the case of split brain, while the groups splits into two, two singleton services run (this is expected). However, if the groups merge, we expected one of the singleton services to stop which wasn't working as expected. Rarely did one of the singleton service stop. In simple terms, we had two singleton services running in a merged setup. Is this a known issue? if yes, do we have a workaround.
PS: We used the default standalone-full-ha profile and we didn't modify the cache properties.
That shouldn't be. Upon merge, a singleton master re-election should take place, and any master should stop.
The workaround for this is to preempt the multiple master on split by requiring a quorum when building your singleton service.