1 2 Previous Next 18 Replies Latest reply on Oct 10, 2006 11:56 AM by Tim Fox

    Once and only once delivery?

    Tim Fox Master

      If I send a reliable message to a server, and it gets added to the queue on the server and persisted, but the server fails before the response is written back to the client, resulting in the client getting an exception even though the message was added successfully to the queue.

      A consumer then consumes the message from the queue, and acknowledges it.

      The client has now caught the exception and it assumes the send failed so it resends the message again, the message gets consumed by the client, resulting in the client having consumed the message twice.

      I.e. we have broken the once and only once delivery guarantee.

      The only way around this AFAICT is to send all messages 2PC, and make the commit on the 2PC transaction idempotent.

      What am I missing here? Am I losing my mind?

        • 1. Re: Once and only once delivery?
          Ovidiu Feodorov Master

          When you say "send a reliable message to a server" you mean sending it in a JMS transaction?

          Because in this case, you should be fine, if the transaction fails for whatever reasons, the message is not written in the DB.

          Is this what you meant?

          • 2. Re: Once and only once delivery?
            Tim Fox Master

            No, just sending a persistent message.

            I don't think a transaction helps anyway. If the failure occurs after the messeg has been persisted in the db, then we still have the same problem.

            • 3. Re: Once and only once delivery?
              Tim Fox Master

              I suppose "once and only once delivery" only really applies to the *delivery* of messages from a queue (or durable sub).

              It doesn't apply to sending a message to a queue.

              However, if it's not possible to write a client that can guarantee it hasn't sent the same message more than once to the same queue, then I'm trying to understand how useful that is.

              So here's the problem:

              Write a JMS client that sends 10 and only 10 messages to a particular queue on a remote server, with a client that consumes those 10, and only those 10 messages.

              Can anyone volunteer a solution?

              Right now, I can't see how that is possible in JMS without using 2PC.

              • 4. Re: Once and only once delivery?
                Elias Ross Master

                Although you suggest 2PC, you could always have the server track the message by ID. If it sees the same message again (and it was successfully sent) then ignore it.

                I'm pretty sure this is something JBossMQ doesn't do.

                • 5. Re: Once and only once delivery?
                  Aaron Walker Newbie


                  When tracking messages on the server side you really need to consider the amount of memory required when you have large message volumes.

                  Also when clustered you have to replicate this information since the client may re-send the message on another node.


                  It might be a better solution for the client to keep track and only for the life of their session. This could be done with a client side interceptor.

                  • 6. Re: Once and only once delivery?
                    Elias Ross Master

                    But then, what's the use of tracking on the client side when the client doesn't know the message did arrive successfully?

                    I've had to deal with billable message traffic coming from an SMPP interface. Ultimately, to prevent the system from billing twice, the client had to supply a unique message ID. This ID was tracked in a central database.

                    Something like this could be easily managed by a JBossCache instance, which would manage replication and expiration of old data. It might be only useful to a few users, and only useful if the client didn't have a transaction going.

                    A more common issue than the JMS client reporting error, is that the interface which generates the JMS traffic (such as an SMPP or HTTP interface for instance), fails to send back "success" and the remote system reattempts the same SMPP/HTTP/etc. transaction again.

                    • 7. Re: Once and only once delivery?
                      Elias Ross Master

                      i would like to add, this common case *would* be solved with a client side interceptor. But then again, with HTTP load balancers, etc. you still need to replicate the request ID on all nodes.

                      Anyway, this feature would be relatively easy to write and might prove useful for a few customers. I don't know what JMS provider (if any) support duplicate detection on MessageProducer.send().

                      • 8. Detection of duplicate messages
                        Tim Fox Master

                        If we want duplicate detection to work between failures of the jms client then we need to do it on the server side.

                        It also needs to work between failures of the server so needs to be persisted in persistent storage.

                        This is certainly going to be a performance drag so I suggest we add a feature request for it, and it to be off by default.

                        We should probably implement it by keeping a cache of message or transaction id. The entries time out and are evicted after a certain amount of time which can be specified by the client.

                        As genman has pointed out, this is not a common feature in messaging systems so I am not too worried if we don't implement this immediately.

                        • 10. Re: Once and only once delivery?
                          Aaron Walker Newbie

                          If you only wanted this feature for persisted messages then you could change the client receive not to actually remove the message from the DB but flag it as being read. Then the you could just check if the message already existed before writing it to the DB. You could then just periodcally sweep and delete the read messages

                          Just a thought......

                          • 11. Re: Once and only once delivery?
                            Tim Fox Master

                            I suspect this would make sense to only implement for persistent messages, since duplicate detection would be an extra QoS over and above that provided by persistent messages.

                            The sweeping is an interesting although you only really want to retain the ids, not the entire message or message reference so this could be wasteful in terms of storage.

                            • 12. Re: Once and only once delivery?
                              Adrian Brock Master

                              I don't understand what you are trying to do?

                              To solve the original problem, you create a transacted session.

                              It is certainly true that for a local tx, you always have the problem
                              of the "transaction observer".

                              1) create transacted session
                              2) send 10 messages in the session
                              3) commit the session

                              The problem being that (3) can work on the server but the network
                              connection fails just at this point, so the client gets a JMSException
                              rather than the commited ok message.

                              Using 2PC certainly solves the problem, because you can prepare()
                              and once you get the ok from that, you are done.

                              If the commit fails (network connection),
                              you can always retry it later using recover().

                              Tracking messages ids isn't going to work, unless you
                              are prepared to hold all messages that were ever sent.

                              1) Client send 10 messages with some identitier on them
                              (commit seems to fail because of network connection).
                              2) Server deliver 10 messages to receiver which are then
                              ACK and removed from the store.
                              3) Client resend 10 messages with the same identiter
                              4) Server happily resends them because it no longer has
                              the original 10 in its store.

                              • 13. Re: Once and only once delivery?
                                Adrian Brock Master

                                In short, if you want this semantic use 2PC.
                                There is no point building something complicated which is eventually
                                going to boil down to something more intensive than tranaction logging
                                anyway!

                                • 14. Re: Once and only once delivery?
                                  Tim Fox Master

                                   

                                  "adrian@jboss.org" wrote:
                                  I don't understand what you are trying to do?

                                  To solve the original problem, you create a transacted session.


                                  A local jms transaction doesn't work for the reasons I mentioned earlier in the thread, and which you mention here:



                                  It is certainly true that for a local tx, you always have the problem
                                  of the "transaction observer".

                                  1) create transacted session
                                  2) send 10 messages in the session
                                  3) commit the session

                                  The problem being that (3) can work on the server but the network
                                  connection fails just at this point, so the client gets a JMSException
                                  rather than the commited ok message.



                                  Tracking messages ids isn't going to work, unless you
                                  are prepared to hold all messages that were ever sent.


                                  This is why I suggested having an expiry on them, so you only keep the ids for a maximum period of time.

                                  Obviously this means that you can send duplicates of messages that were sent before that time, but as you point out this solution won't work 100% of the time unless you have an infinite amount of storage.



                                  1 2 Previous Next