cluster connection/reconnection

[jmesnil]

I've been working on cluster nodes connection / reconnection today.

Discovery case:

when a node use discovery, it usually awaits to receive a UDP broadcast to connect to another nodes and joins the cluster.

However for the 1st server, it is possible it will never receive a UDP broadcast if no other servers is started before its discoveryInitialTimeoutWait expires.

Or when a node becomes the last node of a cluster, it must again listen to UDP broadcast to be notified when another node is UP.

If that happens, we check when the server locator of its cluster connections is notified of a UDP broadcast (ServerLocatorImpl.connectorsChanged()).

If the node has not received any topology, we then call its connect() method to connect to the broadcasting node and triggers the cluster formation.

We need to make sure that the server locator knows when it is part of a cluster (it has received a topology) or when it is alone (it is the only member of the topology). If it is already part of a cluster, we do *not* connect to other broadcasting node (as we are informed of the cluster topology through the connection).

Static connector case:

If a node has a static list of connectors and is notified that a node is down, it creates a runnable which will try to reconnect to the node down.

When the node is UP again, the reconnection will trigger the node notification so that the node UP will be part of the cluster again.

It kind of works but the code is ugly and needs to be cleaned up. I create session factory to connect to other nodes and trigger the node notifications but the factories are not properly handled (when should they be closed?)

I also creates Runnable inside the ServerLocator in the static connector case to try infinitely to reconnect to the static connector list. The runnable should not interfer with the created resources  (esp. when the server locator is on a client side)

The use case are not 100% clear in my mind. I'll clarify them to be sure we handle correctly all cases (discovery/static, first node, last node, etc.)

[timfox]

Jeff Mesnil wrote:

I've been working on cluster nodes connection / reconnection today.

Discovery case:

when a node use discovery, it usually awaits to receive a UDP broadcast to connect to another nodes and joins the cluster.

However for the 1st server, it is possible it will never receive a UDP broadcast if no other servers is started before its discoveryInitialTimeoutWait expires.

Or when a node becomes the last node of a cluster, it must again listen to UDP broadcast to be notified when another node is UP.

You say the node "awaits" - do you mean the node doesn't finish starting up? That shouldn't be the case.

Joining the cluster should never prevent a node starting up and becoming operational. In the UDP case the discovery group maintains its own thread. When a broadcast is received that can prompt the initial connect to another node. But nothing should "wait".

Or when a node becomes the last node of a cluster, it must again listen to UDP broadcast to be notified when another node is UP.

The discovery group never stops listening. It's always listening to broadcasts which it uses to update it's list of initial connectors.

Jeff Mesnil wrote:

If a node has a static list of connectors and is notified that a node is down, it creates a runnable which will try to reconnect to the node down.

When the node is UP again, the reconnection will trigger the node notification so that the node UP will be part of the cluster again.

I'm not sure I understand this. Are you talking abou the *initial* connection or subsequent connections? It's not clear.

If you're talking about a node which is using a static list to make the *initial* connection then, in the case that no other nodes are up, it will try each element in the static list in turn according to the configuration of reconnect-attempts. E.g. if reconnect-attempts = 10 then it will try 10 times to connect to the first element in the list, then try the second one etc. I believe this code already exists.

Once the node has connected to at least one other node in the cluster there is no need for a Runnable. The node will simply receive node UP or DOWN events on the connect and connect to or close cluster connections as appropriate. It's only when the last connection is closed the Runnable is needed.

In the UDP case it's even simpler since the discovery group already has its own thread.

Please don't be scared to discuss this on IRC

It's much better to discuss this if you have doubts than to start developing something which is wrong!

[timfox]

Jeff Mesnil wrote:

We need to make sure that the server locator knows when it is part of a cluster (it has received a topology) or when it is alone (it is the only member of the topology).

I believe that information already exists. When I handed over there was a flag "receivedTopology" in ServerLocatorImpl.

You shouldn't need to create any runnables in ServerLocatorImpl (I remember discussing this before)

[timfox]

I'm looking at the code in ServerLocatorImpl at the point at which I handed over this work.

In the createSessionFactoryMethod you can see:

1) If it's using UDP discovery then it will wait for at least one node in the cluster to broadcast

2) Next thing, it makes a connection either using initial connectors or using topology depending on whether it's receive topology or not.

3) Once it create a connection it waits to receive topology if it's the first connection.

[jmesnil]

let's take the simplest case:

a server is started. It has 1 cluster connection configured with discovery.

When the server starts, its cluster connection starts and in a thread calls its serverLocator.connect().

In this method, the server locator will wait for an UDP broadcast but it won't receive any.

We end up with this server, #0, started and its cluster conn has no initial connectors.

Some time later, a second server, #1, is started. It is also configured with 1 cluster conn with discovery.

When the broadcast group of #1 is received, it will broadcast a UDP packet.

Server #0 will receive this broadcast, use it to set its initial connect *and* trigger the node UP notification ending up with the bridge creation from #0 to #1

The same process happens on server #1 when it receives a UDP broadcast from #0 and ends up with the bridge creation from #1 to #0.

My point is that in the discovery case, if we received a UDP broadcast and the server locator has not received any topology, we trigger the node notification from the discovery entries.

If the server locator has received the topology, we update the initial connectors with the discovery entries but we do *not* trigger the node notification (as it is handled through the connection).

[timfox]

Jeff Mesnil wrote:

My point is that in the discovery case, if we received a UDP broadcast and the server locator has not received any topology, we trigger the node notification from the discovery entries.

If the server locator has received the topology, we update the initial connectors with the discovery entries but we do *not* trigger the node notification (as it is handled through the connection).

No, you always trigger the node notification from the topology, never from the initial list (discovery entries).

In the case where a node is making it's first connection what happens is this:

1) Node receives UDP broadcast, this causes ServerLocator.getConnection() to be called. (to make the first connection).

2) ServerLocator.getConnection() will block until topology is received (since the receivedTopology flag is false)

3) Topology is received from the connection, this causes node UP to be triggered locally

4) Bridge connections are received

Topology events are always received via the connection. I believe the code for this already existed before handover.

Re-iterating the key point here. UDP is *only* used for providing the initial list of connectors. It is not used after that (unless all connections are closed again).