This content has been marked as final. Show 3 replies
Hi, can you give the wiki link here? Thanks
Server Side Failover - It's good to have a buddy watch your back
Every peer(node) has a buddy that will pick up all of it's messages if it fails. If a server fails, the buddy will look in the database and recover all of it's buddies messages. This only works for persistent messages. From the docs - "If the node you are connected to fails, you will automatically fail over to another node and will not lose any persistent messages. You can carry on with your session seamlessly where you left off. Once and only once delivery of persistent messages is respected at all times."
There are two drawbacks to the buddy system.
1. Messages will be stranded if two buddies go down at the same time.
If two nodes go down that are buddies at the same time, one of the buddies messages will not be picked up. For instance, if you have nodes 1, 2, and 3. If 3 is a failover node for 2 and 1 is a failovernode for 3, if 2 and 3 goe down, 2's messages will be lost, becuase 3 is not there to load 2's messages. The messages in Queue 2 will be stranded.
2. Clustered queues will only process messages from running nodes.
When running multiple nodes, the nodes need to be up at the same time to register with one another in the cluster. If you bring the nodes up and down at different times, the messages will not roll over from node to node. For example. Bring node 1 up and process messages. Bring node 1 down. Bring node 2 up and process messages. You will notice that no messages have been pulled from node 1, until that node is brought up. Even though the nodes hit the same database, they will only pull the messages for their own peer id, unless they are registered as a failover node for another peer and failover occurs.
In that case, I think if node 2 starts back up, its message will be restored, if node 3 starts back up, node 2's messages won't be recovered.