Changes to failover functionality
The most complex part of JBM currently is the failover functionality - both client side and server side.
Historically this has been the largest source of bugs and problems to us.
I have thought about this extensively over XMas and would like to propose the following changes which should
a) Simplify our code significantly - making it easier to maintain and reducing defect potential.
b) Making it simpler to understand and configure for the user
Going ahead I think this will add up to a better experience for both us and the customer.
Some of this is already known, and is pretty much what we have alredy discussed, other parts diverge.
As discussed, we will support two different types of deployment scenario for HA:
1) Failover via a shared filesystem:
In this mode, the user configures a shared filesystem (e.g. SAN) that is accessible by each node in the cluster. Each node stores its persistent storage (e.g. BDB environment) there.
When a server fails, it relinquishes an exclusive lock on the file, and another server can start up and load the file.
On the client side connection failure is detected. Since we're not replicating session state any sessions need to be rolled back. But this shouldn't cause a problem to the user if they're using transactional sessions.
This mode will provide the highest level of performance (since no replication required), but trades off against a somewhat slower failover since the new server needs to reload the queue state from the storage.
2) Live-backup replication.
In this mode, a shared FS is not available so we replicate queue state to a backup node. Note the backup node does not become active until it's primary fails. This means failover is fully transparent, and very quick (should be subsecond) since we don't have to reload any state. We also replicate all consumer/session/connection creation to the backup.
One of the best things about this approach is we can remove most client side failover functionality, since, for replicated backup, the connection-session-consumers will already exist on the server side so no need to wait.
For shared filesystem failover, the connection-sesion-consumers on the client side need to be thrown away, and the new node won't be available until it loads its queue - the client can keep retrying (with timeout) until it connects.