4 Replies Latest reply on Jan 31, 2013 9:27 AM by jbertram

    Messages being redelivered in a cluster

    beltane

      Hello,

       

      I am trying to figure out what is supposed to happen to a message when a consumer throws an exception, the message is scheduled for redelivery and the hornetq instance handling the message is is shutdown while in a cluster.  In other words:

      Start instance 1.

      Start instance 2.

      Send a message.

      Instance 1 receives the message and the consumer throws an exception.

      Instance 1 schedules the message for redelivery.

      I shut down Instance 1.

       

      At this point, what should happen to the message?  Naively, I expect that the message will be redistributed to instance 2, but this is not what happens.  Instead, if I restart instance 1, I find the message in the dead letter queue.  Why is the message not redistributed?

       

      Hornetq 2.2.14 running in Tomcat 7.0.27

      The message is sent to a durable queue.

      Redistribution is set to 0.

      Redelivery is set to 5 minutes and it doesn't take me 5 minutes to shut down instance 1.

       

      Thank you for your help!

        • 1. Re: Messages being redelivered in a cluster
          jbertram

          Are instance 1 and instance 2 simply clustered together (i.e. to achieve load-balancing) or are they configured as a live/backup server pair (i.e. to achive HA functionality)?  If they aren't configured as a live/backup server pair then when you shutdown instance 1 all persistent messages which were on that instance will stay there until you restart it.  A node that is down cannot perform any kind of redistribution.

           

          What are your max delivery attempts?

          • 2. Re: Messages being redelivered in a cluster
            beltane

            Hello,

             

            They are clustered together and ha is set to true on ConnectionFactoryConfigurationImpl.  If one instance is shutdown then the other will take any new messages that come in.  The only missing ones are those scheduled for redelivery.

             

            At the moment, for testing, I have the redelivery set to only 1 attempt, just because it is easier for me to keep track of what is occuring.

             

            It is a simple cluster.  I think you are suggesting I should following something like what is at http://docs.jboss.org/hornetq/2.2.14.Final/user-manual/en/html/ha.html and set up a live/backup pair to get the redelivery of durable messages and the live and backup will have a shared journal.

             

            After what you've said, I suspect that I'm only seeing this issue for the messages scheduled for redelivery because I don't have any other messages that have been added, but haven't been consumed.

             

            Thanks.

            • 3. Re: Messages being redelivered in a cluster
              ataylor

              yes, thats exactly what you need to do. Its a coincidence that the messages are scheduled, any messages currently in the crashed server will only be available once the server is restarted or if you configure a backup server.

              • 4. Re: Messages being redelivered in a cluster
                jbertram

                They are clustered together and ha is set to true on ConnectionFactoryConfigurationImpl.  If one instance is shutdown then the other will take any new messages that come in.  The only missing ones are those scheduled for redelivery.

                Unless you have configured HornetQ to actually be HA (i.e. a live/backup pair) it doesn't matter if you set ha to true. 

                 

                I think you are suggesting I should following something like what is at http://docs.jboss.org/hornetq/2.2.14.Final/user-manual/en/html/ha.html and set up a live/backup pair to get the redelivery of durable messages and the live and backup will have a shared journal.

                As Andy said, that is exactly what you should do if you want HA functionality.  You won't get HA functionality if you don't configure the nodes appropriately.