JBoss Messaging - Issues with client side message buffering.
timfox Oct 23, 2005 6:20 AMIn the current mode of operation, we asynchonously shift messages from a destination to the client side buffer of the consumer, so that when a receive() is called on the consumer, a message is possibly already available in the client side buffer, preventing us having to do a server side call - (i.e. this is an optimisation.)
Unfortunately this also brings undesirable side effects - a client can open a consumer and not actually receive() the message (or take a long time between receives) resulting in the message(s) sitting in the client side buffer, potentially for ever.
While messages are in the client side buffer the server considers them as being in the process of delivery and will not deliver them to another consumer if it attaches to the destination.
This can result in messages never being delivered for even well behaved jms clients.
(Aside - Currently we do not actively NACK a message. Messages are only implicitly NACKed when a consumer is closed, at which point any messages in the "being delivered" state revert back into the "available to be delivered" state.)
Possibly solutions to this are as follows:
1. Make every call to receive result in a synchronous (possibly remote) call to the server to receive the "next" message.
This would be a drastic solution, that would involve more network overhead, plus scaling problems with the number of threads in the server performing concurrent receive() operations.
We could enable the above the "default" operation to give spec compliant behaviour, and have the current behaviour as a jboss specific option.
I'm not sure how desirable that would be.
2. Put a internal "jboss-time-to-live" (do not confuse with jms time-to-live) on the message in the client side consumer buffer. When receive() is called on the consumer, the local buffer is inspected for messages, any messages with an expired jboss-time-to-live are discarded and not delivered.
The return value of calls from the server to the consumer client to asynchronously deliver messages could be piggy backed with the ids of messages for which delivery has started in order to prevent the server then timing them out and making the eligible for redelivery simply because the client was taking a long time to process them.
(We would also have to ping consumers from the server even if there were no messages to be sent to consumers in order to find out any more messages that were being processed)
Similarly on the server side, messages that have been sent to the client side consumer buffers for delivery are marked with a similar jboss-time-to-live, preventing them from being re-delivered while the jboss-time-to-live is not expired.
Difficulties with this method are in dealing with differences in clocks between different machines in order to synchronize time-to-live values - which could result in messages being delivered more than once -this could be compensated with by making sure the client side "time-to-live" is always less than the server side "time-to-live" by a safety factor to allow for network latency and difference in clock speeds. The complexity of this could make this one a difficult one to implement and still retain once and only once semantics.
3. A better solution.....
Comments please.