Multiple backups failover with shared store

[jmesnil]

I don't understand how multiple backups are supposed to work in the shared store configuration.

In tim's original design, when using shared store, the backups all have a single reference to a live connector (<live-connector-ref in their configuration).

Let's take the following use case:

* node #0 is live

* nodes #1, #2, and #3 are configured as backup with a <live-connector-ref> to node #0.

1. node #0 is started

=> it becomes live

2. node #1 is started

=> it takes the backup.lock, becomes the active backup and connects to node #0 to update its topology

3. node #2 and #3 are started

=> they are both waiting to take the backup.lock

Then, node #0 crashes

1. node #1 takes the live.lock and becomes live

2. node #2 takes the backup.lock, becomes the active backup and connects to node #0

=> but node #0 is no longer the live node, and node #2 will not connect to the new live node (#1) to update its topology.

I don't see how this could work. If we have multiple backups, they must be prepared to have any live node and not necessarily the one which was

expected at configuration time.

Andy and I updated the backup node configuration to accept either a static list of live server or a discovery group.

This way, when a node becomes an active backup (taking the backup.lock), it has an opportunity to find which node is the current live node.

How does it find it?

Using a static connector list, the backup will iterate on each connector configuration and try to open a connection (with only 1 attempt).

It will do this repeatedly until one of the connection is opened => this is the live node.

Using a discovery group, the backup will wait for a UPD packet from the live node to know it.

=> This implies that the backup (and the live server) must all have broadcast group and discovery group dedicated to live/backup discovery.

Could it be the same ones that are used for clustering?

I am focusing on the static list.

When a backup node becomes the main backup (after taking the backup.lock), it starts its cluster manager and tries to find the live connector.

Based on the static list of connectors, it try each one and connect to it with a single attempt.

We don't use infinite attempt to give each connector a chance but we do this infinitely until one connector replies (we could be smarter later and

do this in parallel and join when the connection is opened).

Once the backup has a connection to the live node, it will subscribe to its topology and announce its own presence to the live node (so that this one

could inform its topology listeners of the backup node)

I'll be working on this today.

I don't understand how multiple backups are supposed to work in the shared store configuration.

In tim's original design, when using shared store, the backups all have a single reference to a live connector (<live-connector-ref in their configuration).

Let's take the following use case:

* node #0 is live

* nodes #1, #2, and #3 are configured as backup with a <live-connector-ref> to node #0.

1. node #0 is started

=> it becomes live

2. node #1 is started

=> it takes the backup.lock, becomes the active backup and connects to node #0 to update its topology

3. node #2 and #3 are started

=> they are both waiting to take the backup.lock

Then, node #0 crashes

1. node #1 takes the live.lock and becomes live

2. node #2 takes the backup.lock, becomes the active backup and connects to node #0

=> but node #0 is no longer the live node, and node #2 will not connect to the new live node (#1) to update its topology.

I don't see how this could work. If we have multiple backups, they must be prepared to have any live node and not necessarily the one which was

expected at configuration time.

Andy and I updated the backup node configuration to accept either a static list of live server or a discovery group.

This way, when a node becomes an active backup (taking the backup.lock), it has an opportunity to find which node is the current live node.

How does it find it?

Using a static connector list, the backup will iterate on each connector configuration and try to open a connection (with only 1 attempt).

It will do this repeatedly until one of the connection is opened => this is the live node.

Using a discovery group, the backup will wait for a UPD packet from the live node to know it.

=> This implies that the backup (and the live server) must all have broadcast group and discovery group dedicated to live/backup discovery.

Could it be the same ones that are used for clustering?

I am focusing on the static list.

When a backup node becomes the main backup (after taking the backup.lock), it starts its cluster manager and tries to find the live connector.

Based on the static list of connectors, it try each one and connect to it with a single attempt.

We don't use infinite attempt to give each connector a chance but we do this infinitely until one connector replies (we could be smarter later and

do this in parallel and join when the connection is opened).

Once the backup has a connection to the live node, it will subscribe to its topology and announce its own presence to the live node (so that this one

could inform its topology listeners of the backup node)

I'll be working on this today.