2 Replies Latest reply on Dec 19, 2008 10:50 AM by clebert.suconic

    Failover on LargeMessages

    clebert.suconic

      When sending LargeMessages (server2client), I need to have the masterNode inform the backupNode what's happening on the sending.

      The process of sending small messages on clustering seems simple to me, as you can do it atomically. You inform the backup about sending the message, and the backup will place it on the buffer. And receiving the confirmation on flow-control is also simple.


      However on largeMessages this is a little bit different:

      I - I inform the backup node that I'm starting a MessageSend, so the backup-node will take out the delivery from the queue, and create the LargeMessages.

      II - I can't just let the backup-node call largeMessageSender.send, or the backup could be running faster than the server. (Running out of Credits first for instance).

      So I'm currently implementing it in a way the live-node is aways informing the backup-node what position of the LargeMessage is being sent to the client. In case of failure the backup-node will resume from that chunk forward. (of course using the logic that is already in place for packet confirmation).

      Another option I have is reset everything and start sending large message again from its beginning in case of failure. I will check that option tomorrow.

        • 1. Re: Failover on LargeMessages
          timfox

          Ok, before jumping in with a solution, could you explain a bit more what the problem is (the diagnosis)?

          Your post really just explains your solution, not the underlying issue.

          • 2. Re: Failover on LargeMessages
            clebert.suconic

             

            II - I can't just let the backup-node call largeMessageSender.send, or the backup could be running faster than the server. (Running out of Credits first for instance).



            This is the main issue.

            When I send a message from Server2Client, I will verify that is a LargeMessage, and start a loop on sending the several chunks.

            If I just call largeMessageSender.send on the backup, I could the backup flooding flowControl before the liveNode. A lot of uncertain events could happen followed after that.


            Example...
            - Master will send an 1G Message.
            - It will replicate the send to the backup
            - Backup will start sending the 1G Message. It will immediately flood flow control. (There is no actual sending on backup).
            - Master will receive credits and replicate them to backup
            - Master will send another 1G Message
            (Eventually, the backup will still be sending the previous message, I have seen situations where the flowControl still full on backup, and that will reject replications from the masterNode).


            We need a sync between Master and Backup to solve that. Or I implement that sync chunk by chunk, and failover would resume from that chunk, Or I sync just a start and end, and failover would reset the pointers and file and resume from the beggining of the file again.