Does hornetq let you acknowledge from a different thread?

[steffi]

If you have a pattern where you have async consumption thru message listener but you perform that actual processing in a worker thread is it possible to acknowledge the message in the worker thread with hornetq?

This is something JMS generally doesn't allow you to do and so you have a possibility of work still be queued for work in a thread pool and having already acknowledged the message from the queue. In the event of an interruption it's possible to have lost messages. In order to ensure that I can process my message in a thread pool and I have maintain state in a database which is an additional cost that I have per message and one I'm not comfortable with.

I would much prefer a solution that allows me to perform my work in a different thread but acknowledge the message only when I know I've processed.

[timfox]

Why would you want to do this? What's the advantage of processing your message with a different thread?

[steffi]

In my case I'm pulling the message of the queue asynchronously via message listener and then distributing it for processing to a threadpool. Doing this maintain the order of processing. meaning that threads in the thread pool each have queues and based on characteristics of the message related messages all get assigned to the same thread or at least the same queue so that order of messages by this characteristic is maintained. I only know when I've processed a message once it's been processed by a thread from this thread pool and not the async delivery thread so the message is immediately acknowledged prior to being dispatched. Currently there's a window where after acknowledgement you could lose messages so I have to maintain state in a database so that I can recover if there's an interruption. This state comes with a very high cost. I remove this state when I've finished processing the message in the worker thread.

messages that I pull off the queue have some key and the requirement is that all messages need to processed chronologically by this key but because I use a thread pool I can parallelize processing of all messages in groups of this key.

[timfox]

It would be illegal to do this using JMS, since in JMS a session is single threaded - you cannot send messages to be processed on different threads. This may be appear to work with some messaging systems but you are likely to get spurious failures and other weird stuff happening if you do this.

Similarly on the core API a session is single threaded.

If you want to parallelise consumption of messages the correct way of doing this in either JMS or the core API is to create multiple sessions and consume messages using these from the same queue.

[steffi]

How fast relative to MQ is your "selector" implementation in hornetq? Since that's how I'd maintain ordering across multiple consumers with that approach since messages for the same group would have to be processed by the same consumer.

[ataylor]

"steffi2" wrote:
How fast relative to MQ is your "selector" implementation in hornetq? Since that's how I'd maintain ordering across multiple consumers with that approach since messages for the same group would have to be processed by the same consumer.

Well we are only in Beta at the moment so as yet we have no figures. This will happen after GA. What i can say is that we are pretty quick, each consumer with a message selector has its own iterator on the queue so we can deliver pretty fast, this is a nice piece of code that Jeff wrote.

Alsdo you have you looked at the strict ordering (grouping functionality). This pretty much ties messages with a specific group id to a single consumer. take a look at the message-grouping example and the user manual.

[timfox]

"ataylor" wrote:

"steffi2" wrote:
How fast relative to MQ is your "selector" implementation in hornetq? Since that's how I'd maintain ordering across multiple consumers with that approach since messages for the same group would have to be processed by the same consumer.

Well we are only in Beta at the moment so as yet we have no figures. This will happen after GA. What i can say is that we are pretty quick, each consumer with a message selector has its own iterator on the queue so we can deliver pretty fast, this is a nice piece of code that Jeff wrote.

Alsdo you have you looked at the strict ordering (grouping functionality). This pretty much ties messages with a specific group id to a single consumer. take a look at the message-grouping example and the user manual.

It would be very easy to test the speed of a message selector.

Simply create a SelectorImpl in the code and call it in a loop. About 5 lines of code.

However, message grouping seems more appropriate in this case than selectors, as Andy says