Once and only once delivery?

[timfox]

If I send a reliable message to a server, and it gets added to the queue on the server and persisted, but the server fails before the response is written back to the client, resulting in the client getting an exception even though the message was added successfully to the queue.

A consumer then consumes the message from the queue, and acknowledges it.

The client has now caught the exception and it assumes the send failed so it resends the message again, the message gets consumed by the client, resulting in the client having consumed the message twice.

I.e. we have broken the once and only once delivery guarantee.

The only way around this AFAICT is to send all messages 2PC, and make the commit on the 2PC transaction idempotent.

What am I missing here? Am I losing my mind?

[ovidiu.feodorov]

When you say "send a reliable message to a server" you mean sending it in a JMS transaction?

Because in this case, you should be fine, if the transaction fails for whatever reasons, the message is not written in the DB.

Is this what you meant?

[timfox]

No, just sending a persistent message.

I don't think a transaction helps anyway. If the failure occurs after the messeg has been persisted in the db, then we still have the same problem.

[timfox]

I suppose "once and only once delivery" only really applies to the *delivery* of messages from a queue (or durable sub).

It doesn't apply to sending a message to a queue.

However, if it's not possible to write a client that can guarantee it hasn't sent the same message more than once to the same queue, then I'm trying to understand how useful that is.

So here's the problem:

Write a JMS client that sends 10 and only 10 messages to a particular queue on a remote server, with a client that consumes those 10, and only those 10 messages.

Can anyone volunteer a solution?

Right now, I can't see how that is possible in JMS without using 2PC.

[genman]

Although you suggest 2PC, you could always have the server track the message by ID. If it sees the same message again (and it was successfully sent) then ignore it.

I'm pretty sure this is something JBossMQ doesn't do.

[apwalker]

When tracking messages on the server side you really need to consider the amount of memory required when you have large message volumes.

Also when clustered you have to replicate this information since the client may re-send the message on another node.


It might be a better solution for the client to keep track and only for the life of their session. This could be done with a client side interceptor.

[genman]

But then, what's the use of tracking on the client side when the client doesn't know the message did arrive successfully?

I've had to deal with billable message traffic coming from an SMPP interface. Ultimately, to prevent the system from billing twice, the client had to supply a unique message ID. This ID was tracked in a central database.

Something like this could be easily managed by a JBossCache instance, which would manage replication and expiration of old data. It might be only useful to a few users, and only useful if the client didn't have a transaction going.

A more common issue than the JMS client reporting error, is that the interface which generates the JMS traffic (such as an SMPP or HTTP interface for instance), fails to send back "success" and the remote system reattempts the same SMPP/HTTP/etc. transaction again.

[genman]

i would like to add, this common case *would* be solved with a client side interceptor. But then again, with HTTP load balancers, etc. you still need to replicate the request ID on all nodes.

Anyway, this feature would be relatively easy to write and might prove useful for a few customers. I don't know what JMS provider (if any) support duplicate detection on MessageProducer.send().

[timfox]

If we want duplicate detection to work between failures of the jms client then we need to do it on the server side.

It also needs to work between failures of the server so needs to be persisted in persistent storage.

This is certainly going to be a performance drag so I suggest we add a feature request for it, and it to be off by default.

We should probably implement it by keeping a cache of message or transaction id. The entries time out and are evicted after a certain amount of time which can be specified by the client.

As genman has pointed out, this is not a common feature in messaging systems so I am not too worried if we don't implement this immediately.

[timfox]

Added task http://jira.jboss.com/jira/browse/JBMESSAGING-604

[apwalker]

If you only wanted this feature for persisted messages then you could change the client receive not to actually remove the message from the DB but flag it as being read. Then the you could just check if the message already existed before writing it to the DB. You could then just periodcally sweep and delete the read messages

Just a thought......

[timfox]

I suspect this would make sense to only implement for persistent messages, since duplicate detection would be an extra QoS over and above that provided by persistent messages.

The sweeping is an interesting although you only really want to retain the ids, not the entire message or message reference so this could be wasteful in terms of storage.

[adrian.brock]

I don't understand what you are trying to do?

To solve the original problem, you create a transacted session.

It is certainly true that for a local tx, you always have the problem
of the "transaction observer".

1) create transacted session
2) send 10 messages in the session
3) commit the session

The problem being that (3) can work on the server but the network
connection fails just at this point, so the client gets a JMSException
rather than the commited ok message.

Using 2PC certainly solves the problem, because you can prepare()
and once you get the ok from that, you are done.

If the commit fails (network connection),
you can always retry it later using recover().

Tracking messages ids isn't going to work, unless you
are prepared to hold all messages that were ever sent.

1) Client send 10 messages with some identitier on them
(commit seems to fail because of network connection).
2) Server deliver 10 messages to receiver which are then
ACK and removed from the store.
3) Client resend 10 messages with the same identiter
4) Server happily resends them because it no longer has
the original 10 in its store.

[adrian.brock]

In short, if you want this semantic use 2PC.
There is no point building something complicated which is eventually
going to boil down to something more intensive than tranaction logging
anyway!

[timfox]

I don't understand what you are trying to do?

To solve the original problem, you create a transacted session.

A local jms transaction doesn't work for the reasons I mentioned earlier in the thread, and which you mention here:

It is certainly true that for a local tx, you always have the problem
of the "transaction observer".

1) create transacted session
2) send 10 messages in the session
3) commit the session

The problem being that (3) can work on the server but the network
connection fails just at this point, so the client gets a JMSException
rather than the commited ok message.

Tracking messages ids isn't going to work, unless you
are prepared to hold all messages that were ever sent.

This is why I suggested having an expiry on them, so you only keep the ids for a maximum period of time.

Obviously this means that you can send duplicates of messages that were sent before that time, but as you point out this solution won't work 100% of the time unless you have an infinite amount of storage.