3 Replies Latest reply on Jan 29, 2007 8:21 PM by clebert.suconic

    Failover possible dead lock

    clebert.suconic
    clebert.suconic Jan 29, 2007 6:53 PM

      I found a possible dead lock on Failover / valves.

      This is the scenario:


      One Connection

      Two Threads
      .. Thread1 is called Consumer
      .. Thread2 is called Producer

      (Each thread will have its own session, of course)


      The Consumer thread will do:

      
 while (true)
 {
 Message message = consumer.receive();
 System.out.println("Message = " + message);
 }



      While the Producer will do 
      while(true)
{
 producer.send(session.createTextMessage("Message from producer " + id + " counter=" + (counter)));
}



      When the server dies.. there is a possibility the server died while consumer.receive is being executed.

      Consumer.receive is not doing any calls to the server, as it is just waiting Callback to come and notify the consumerDelegate. The result is when producer detects the failover it can't close the valve as consumer will never give up its position, and also never detect a failure.

      A solution for this problem would be to ignore the valve interceptor when receive is being executed. Is this a valid solution?

      I found this while debugging MultiThreadFailoverTest (former HAStressTest).

      I could create a testcase for this but the failure would be a dead lock, and the test would eventually fail in timeout.

      • 7856 Views
      • Tags:
        • 1. Re: Failover possible dead lock
          timfox
          timfox Jan 29, 2007 7:02 PM (in response to clebert.suconic)

          I don't think this is a deadlock - deadlock implies a race for two exclusive locks:

          thread 1 gets lock on resource A.

          thread 2 gets lock on resource B

          thread 1 tries to get lock on resource B and blocks

          thread 2 tries to get lock on resource A and blocks

          However, it's still an issue.

          Can't you interrupt the receive thread, to cause it to return?

            • Actions
          • 2. Re: Failover possible dead lock
            clebert.suconic
            clebert.suconic Jan 29, 2007 7:20 PM (in response to clebert.suconic)

             

            "Tim Fox" wrote:
            I don't think this is a deadlock - deadlock implies a race for two exclusive locks:


            Well.. I know technical name wouldn't be dead lock...

            But you will still be waiting forever on a writeLock, that will never occur. (you're dead anyway :-) )


            I guess I could call Failover to notify the receive thread.. yes... but I just don't think it would be a good solution. The way it works now, we only start failure detection after we close the valve. We can't close the valve if there is an invocation pending.

            I would prefer using the Valve only on places where a server communication is needed. On that case we don't have a problem as we will aways get an IOException or a CannotConnectException when the server dies.

            A receive is only waiting on an internal buffer and I don't see any problems on letting it wait Callback to be re-established and feed the buffer again after failover.

            I have talked to Ovidiu by IM, and he thinks it would make sense to use the Valve only on places where an IO to the server is being performed.

              • Actions
            • 3. Re: Failover possible dead lock
              clebert.suconic
              clebert.suconic Jan 29, 2007 8:21 PM (in response to clebert.suconic)

              http://jira.jboss.org/jira/browse/JBMESSAGING-790

                • Actions