4 Replies Latest reply on Feb 13, 2011 3:57 PM by marcus.nonius

    Msg delivery order in queue

    marcus.nonius

      Hi,

       

      although I read many postings, articles etc. about how the (JMS) behaviour should be concerning the message delivery order in queues, I am not quite sure about this; even more confused. So - it's very important to me to know, which guarantees gives HornetQ about this.

       

      Scenario:

      • One durable queue, two single threaded producers P1 and P2, one single threaded consumer C1.
      • Consumer sending persistent and scheduled (property '_HQ_SCHED_DELIVERY') messages to the queue.
        [Detail: The scheduled 'delay' is very short, only System.currentMillis+200millis; going to be used with every msg to compensate latency/jitter]

       

      My understanding is, the delivery order of messages of each producer to C1 has to be accurate the according sending order (FIFO).

      Question: Is this true? And even more also under heavy load, so that possibly the delivery time of the de-paged scheduled messages has already expired?

      I am aware of the fact, that the ordering between messages from P1 and P2 can't be guaranteed.

       

      Please excuse to mention ActiveMQ here : They explicit propagate a feature called 'exclusive consumer' (http://activemq.apache.org/exclusive-consumer.html). Because the ordering is an issue and possibly depends on the JMS provider?!

       

      Many thx in advance for an answer.

      Marcus

        • 1. Msg delivery order in queue
          ataylor

          The first thing to point out here is that the JMS spec in no way covers scheduled delivery of messages so talking about delivery order really makes no sense, you are changing the FIFO constraint by setting a scheduled delivery. Internally we use a Scheduled Executor to do this which uses the system clock so obviously there are no gaurantees thatthis would be accurate.

           

          regarding your exclusive consumer comment i have no idea why you think this is at all related to your problem.

          #

          What is it you are actually trying to acheive?

          • 2. Msg delivery order in queue
            marcus.nonius

            Andy,

            thank you for your answer.

             

            I know that scheduled messages are not part of the JMS spec.

            But shouldn't it be the case, that, if all messages do carry the same scheduling 'offset' (System.currentMillis+200millis), the delivery order to a single consumer nevertheless should be in FIFO order? Or, do I understand you right, can't that not be guaranteed any more because of the use of the internal Scheduled Executor? The other way around: Without the scheduled attribute, the messages would then be guaranteed delivered in FIFO order?

             

            I mentioned the exclusive consumer feature because I am wondering, that this feature is explicit pointed out. Isn't that an indication, that the delivery order is not a matter of course - indepent of scheduled messages (i.e. not supported from every provider)?

             

            What do I try to achieve:

            The purpose of the 200ms scheduled 'offset' is to compensate little network and/or device latency. That's a very important issue in our system at whole. Each message carry an internal used timestamp (system wide synchronized). Multiple producers (devices) are sending to a central queue, processed by an single threaded consumer. Within the 200ms scheduled 'offset' we are able to compensate latency between the internal timestamps, to guarantee an monoton increasing chronologic flow of the messages towards the processing logic.

            That doesn't avoid that messages between two different or multi-threaded producers getting out of chronologic order in edge situations - accepted.  But within one single-threaded producer, the order must not change.

            • 3. Msg delivery order in queue
              ataylor

              I know that scheduled messages are not part of the JMS spec.

              But shouldn't it be the case, that, if all messages do carry the same scheduling 'offset' (System.currentMillis+200millis), the delivery order to a single consumer nevertheless should be in FIFO order? Or, do I understand you right, can't that not be guaranteed any more because of the use of the internal Scheduled Executor? The other way around: Without the scheduled attribute, the messages would then be guaranteed delivered in FIFO order?

              of course its not garanteed as its non deterministic, for a start your assuming that (System.currentMillis+200millis) is the same from every client, obviously this cannot be the case.this is no different from any other messaging providor, maybe you misunderstand some of the concepts about messaging, when using queues we are talking about p2p, i.e. point to point, which is the singular from both ends, that is there is one producer and one consumer.

               

              I mentioned the exclusive consumer feature because I am wondering, that this feature is explicit pointed out. Isn't that an indication, that the delivery order is not a matter of course - indepent of scheduled messages (i.e. not supported from every provider)?

              again, i think you dont understand what P2P is, order is gauranteed between 2 points.

               

              What do I try to achieve:

              The purpose of the 200ms scheduled 'offset' is to compensate little network and/or device latency. That's a very important issue in our system at whole. Each message carry an internal used timestamp (system wide synchronized). Multiple producers (devices) are sending to a central queue, processed by an single threaded consumer. Within the 200ms scheduled 'offset' we are able to compensate latency between the internal timestamps, to guarantee an monoton increasing chronologic flow of the messages towards the processing logic.

              That doesn't avoid that messages between two different or multi-threaded producers getting out of chronologic order in edge situations - accepted.  But within one single-threaded producer, the order must not change.

              So it sounds to me as if you want to gaurantee the order messages are consumed according to the timestamp that they are produced, maybe you need to think about a different technology.

              • 4. Re: Msg delivery order in queue
                marcus.nonius

                Hi,

                 

                yes, I verified it. Without the scheduled attribute the messages are in order (P2P as you mentioned).

                Afterwords it's obvious to me. Thx for the hint !