12 Replies Latest reply on Oct 5, 2007 5:43 AM by timfox

    Messages are moved to expiry queue only when there is a cons

    hendra_netm

      Hi guys,

      I am trying the scenario where one MessageProducer send many messages to a queue and set time to live for each message 5 seconds.

      The expiry queue is set, and I expect after 5 seconds, all messages will be expired and moved to the expiry queue.
      This is happened, only if there is a MessageConsumer registered in the queue. Otherwise, the message will not expire until another consumer is set up for the queue.

      is this a bug or the intended behaviour? I didn't find any documentation about this one, either on JBM documentation or JMS specification.

      If this is intended behaviour, I will always put a dummy consumer as the solution of my problem ;)

      Thank you in advance.

      Best Regards,
      Hendra

        • 1. Re: Messages are moved to expiry queue only when there is a
          hendra_netm

          Forget to mention. I am using JBM 1.3.0 GA and JBoss AS 4.2.0

          • 2. Re: Messages are moved to expiry queue only when there is a
            timfox

            Yes, currently the check to see if a message is expired is only done on delivery. This is sufficient to be JMS spec compliant.

            It would be a fairly intensive operation to continously scour the queues for expired messages.

            However I have added a task for JBM 2.0 to implement this.

            • 3. Re: Messages are moved to expiry queue only when there is a
              genman

              What I did for JBoss MQ is schedule a timeout to occur for all expiring messages, similar to the scheduled message feature, which does the move/removal.

              There are some scalability issues with doing that. Namely, every soon and not-so-soon expired message ID is kept in memory on a heap. It'd be better if the expiry date was kept as a separate column in the DB, and indexed, to assist in finding these messages without having to scan them all.

              • 4. Re: Messages are moved to expiry queue only when there is a
                timfox

                 

                "genman" wrote:
                What I did for JBoss MQ is schedule a timeout to occur for all expiring messages, similar to the scheduled message feature, which does the move/removal.

                There are some scalability issues with doing that. Namely, every soon and not-so-soon expired message ID is kept in memory on a heap. It'd be better if the expiry date was kept as a separate column in the DB, and indexed, to assist in finding these messages without having to scan them all.


                Yes, I thought about this too.

                It would be the easiest solution to implement, but as you mention too, I think it has scalability issues.

                I am thinking along the lines of a expirer thread that periodically scans the queue

                • 5. Re: Messages are moved to expiry queue only when there is a
                  julians37

                  Hi,

                  I've tried to register a dummy MessageConsumer as suggested by hendra_netm, but that doesn't seem to work. If I specify a matching selector, I'm receiving the messages that are meant to be expired. And if I specify a selector like "1 = 0", that doesn't seem to cause expiry to be checked.

                  What seems to work is to unregister and re-register the dummy consumer in regular intervals, but I wonder if there's a more elegant approach - any advice?

                  Thanks,

                  Julian

                  • 6. Re: Messages are moved to expiry queue only when there is a
                    hendra_netm

                    Hi julians37,

                    Mine works fine. I don't use selector or anything. Just create a MessageConsumer for the queue, and start the connection.

                    The MessageConsumer does not call "receive()" method, so it won't receive any message. That's why I called it "dummy".

                    Anyway, how do you set your message expiration?

                    Regards,
                    Hendra

                    • 7. Re: Messages are moved to expiry queue only when there is a
                      julians37

                      Hi Hendra,

                      thanks for your reply. Unfortunately I can't get it to work, my messages seem to get stuck in the dummy MessageConsumer (even though I never call receive() on it). I tried with and without transactions and with all kinds of acknowledgement but no go...

                      I also don't see how it could work, doesn't a MessageConsumer have an internal buffer where it stores incoming messages - otherwise, how could MessageConsumer.receiveNoWait() ever return a message?

                      Maybe it's the way you set up your Connection or your Session, would you mind sending over your code?

                      Any chance that a JBM developer could chime in and let me briefly know whether this is actually meant to work? If it's meant to be that straightforward, why the discussion about queue reapers and scalability?

                      Thanks a lot,

                      Julian

                      • 8. Re: Messages are moved to expiry queue only when there is a
                        julians37

                        Oh and I use StompConnect and the STOMP protocol's "expires" header, which in turn just sets JMSExpiration... is there a different way of setting expiration?

                        • 9. Re: Messages are moved to expiry queue only when there is a
                          timfox

                           

                          "julians37" wrote:

                          Any chance that a JBM developer could chime in and let me briefly know whether this is actually meant to work? If it's meant to be that straightforward, why the discussion about queue reapers and scalability?


                          Julian - could you explain in a bit more detail about what you are trying to achieve?

                          Thx

                          • 10. Re: Messages are moved to expiry queue only when there is a
                            julians37

                            Sure, thanks for your attention.

                            I have a request/response pattern between a client and a server communicating via JBM. The client sends a request and waits for the server to send the corresponding response. Since the request processing on the server side can take a while in some situations, I'd like the client to be able to cancel waiting for the response, and in this case I need the response delivered asynchronously (via e-mail) so that the user knows when the request has been processed.

                            It seems like the expiry queue feature would lend itself to implementing this: if the response doesn't get picked up by the client within a timeout of, say, 2 seconds, it expires and gets transferred to a second queue where it's picked up by a component that takes care of forwarding it by mail.

                            In this scenario it's not good enough for the message to be expired once an attempt is being made to pick it up on the original response queue (as there will possibly be no client listening for responses anytime soon). Instead, I need it to be moved once it's expiring.

                            Does this makes sense or would you consider this to be an abuse of the expiry feature? Do you see a way of implementing this in another way, maybe using temporary topics and a DLQ? (Right now I'm not using temporary topics for responses - instead I use a global response queue with a selector on the JMS_CorrelationID.) Unfortunately there's no way of selecting messages by age or that would be another way of handling this.

                            FWIW, I'm currently working around the issue by having a dummy consumer with a selector that never matches re-registering with the queue periodically, and with this workaround my system works fine. But it's obviously an ugly solution - even though the number of messages involved is negligible so there are no performance issues.

                            Thanks,

                            Julian

                            • 11. Re: Messages are moved to expiry queue only when there is a
                              hendra_netm

                              hi julians37,

                              I just remembered, that you need to set your expire time in send method of message producer

                              send(Destination destination, Message message, int deliveryMode, int priority, long timeToLive)

                              you cannot set your message expiration by using setJMSExpiration. May be you are using this way to set your message expiration??

                              Btw, in my opinion you can use JMS timestamp to select messages by age.

                              Best Regards,
                              Hendra

                              • 12. Re: Messages are moved to expiry queue only when there is a
                                timfox

                                 

                                "julians37" wrote:

                                Does this makes sense or would you consider this to be an abuse of the expiry feature? Do you see a way of implementing this in another way, maybe using temporary topics and a DLQ? (Right now I'm not using temporary topics for responses - instead I use a global response queue with a selector on the JMS_CorrelationID.) Unfortunately there's no way of selecting messages by age or that would be another way of handling this.



                                The only guarantee a JMS compliant messaging system has to make with regard to message expiry is to ensure that you don't ever receive a message after the expiry time, if the expiry time has been specified.

                                JBM is fully compliant in that regard.

                                Anything to do with expiry queues etc, is beyond the scope of the spec.

                                In other words there's no guarantee that a particular JMS provider actually moves the message to the expiry queue actually at, or near to the desired expiry time.

                                You'll probably find that different providers do this in different ways, so designing your application to rely on this is probably a bad move.

                                In future versions of JBM we will probably implement some kind of expiry reaper thread that trawls the queues for expired messages and puts them on the expiry queue (if it is specified), but even with this the exact time they will get to the expiry queue is not really reliable, subject to how often the thread is scheduled to run, number of messages etc.