HASingletoneton Timer
I'm trying to work through standing up deployment of EJB Timer services within in a HA Clustered JBOSS EAP 6.3 environment. I've looked through the quick start (clustered-ha-singleton) and have the basics working fine on a 2 node deployment. The challenge I'm trying to work through is reconfiguration of the timers via JMX (or other means) AND proper behavior if one of the node is shutdown and then brought back on line.
I created the class that implements the ServiceActivator interface and the service (HATimerService) implementation which implements Service<String>. In HATimer service I followed the example and use grabbed a reference to my Timer EJB via a JNDI lookup which I then use to kickoff my timer.
From here I I created both JMX and RESTful access points to reconfigure the EJB Timer service which seems to work bot of which operate against the EJB Timer. I have enabled a JMX on the EJB itself and also injected the EJB into business layer backing the restful service--so there are 2 ways to reconfigure the bean.
But here are the problems...
1). The above seems to work...I'm able to manipulate the settings of the EJB timer as I'd expect however, if I make changes to the timer (say change the timeout periodicity) and then shutdown the node--those changes are not being preserved/replicated to the timer on the other node in my domain cluster.
2) When I kill a node and then bring the node back on line, I now have 2 instances of my timer. The singleton service doesn't detect/respect there is already a copy running in the cluster.
Is there clean way to ensure and retain sync between the timers? Is there a example or sample someone can point to? The quick start suggests using a datasource with a timer to trigger going and checking for configuration updates as a means of reconfiguration. Presumably the intent with the recommendations was the EJB timer would have an internal timer that that would asynchronously fire to force the timer bean to check for configuration changes in some datasource...which could work but it doesn't address the issue encountered when stopping/restarting nodes in the domain.
I haven't yet come across anything in the documentation to encourage a recommended approach for this scenario...any thoughts/opinions would be great!