3 Replies Latest reply on Nov 19, 2009 9:57 AM by clebert.suconic

    IO recursion over callbacks

    clebert.suconic

      Since I'm starting to execute the transactional callbacks from the AIO/NIO callbacks,

      We shouldn't feed back on any journal operation inside those transactional callback operations, or we would eventually dead lock the notification thread.


      I have two places that I have seen this happening so far:

      I - the DuplicateIDImpl on afterCommit
      II - the message delete on afterCommit after processing ACKs.

      Maybe there are others.. but those are what I have seen so far. (and still also what I remember from when i looked at the replication)

      for the DuplicateID I could fix it easily by using an executor. (I still need to validate if that's valid or not).


      I hope I should be able to do the same with the message delete. If not we will need to refactor it somehow.

        • 1. Re: IO recursion over callbacks
          timfox

          I can't really work out from the explanation what you mean here, but, if it helps to clarify things: we only need to wait on completion for some sync operations, they are:

          1) Send of a persistent message, when syncNonTransactional = true

          2) Ack of a persistent message, when syncNonTransactional = true

          3) Commit of a transaction, when syncTransactional = true

          4) Prepare of a transaction, when syncTransactional = true

          5) Rollback of a transaction, when syncTransactional = true

          For all the above operations, there is only one sync, or wait completion, so I don't understand where this "recursion" comes in.

          • 2. Re: IO recursion over callbacks
            timfox

            Don't make it too complex.

            • 3. Re: IO recursion over callbacks
              clebert.suconic


              3) Commit of a transaction, when syncTransactional = true


              Let me take this case to make an example. We can talk over IRC later about this:


              I - TransactionImpl will be responding to an IOcallback. As soon as the commit is back from NIO/AIO it will fire the operations.


              II - The notification thread will then fire a callback. While inside the callback it will perform all the afterCommits.

              III - As soon as you process the ACK, we are deleting the message inside the callback. That will call the journal back, while inside the notification thread.

              IV - The AIO notification are also used internally for things like writing headers on files. As soon as you run out of files, another thread will try to open a file, and write the file header.

              V - At the end you will have:
              - The notification thread firing back an IO operation (e.g deleteMessage). The delete message could be waiting for an opened file.
              - The thread that feeds the opened files will be waiting on the callback for the header, but the callback for the header will never come as it is busy deleting a file.

              That's a dead lock!


              We can't have any IO operations inside the IOCallback. That's a recursivity that will cause dead locks, and we currently have those.