JMS topology (Queue with multiple consumers) and message grouping

[gab-_-]

here is a simplified / schematic topology we set up

       [Front server 1 (message producer)]  [Front server n (message producer)]

                       |                                    |      

                       |________________    ________________|

                                        |  |       

                             [Messaging Server (HornetQ)]

                                        |  |

                        ________________|  |________________

                        |                                   |

       [Task server 1 (Message Driven Bean)]          [Task server n (MDB)]

- Each node (server) is a standalone (no cluster) jboss application server (Jboss-as7) including the messaging one.

- The messaging server deploys many JMS queues.

- Each Task server deploys a MDB per queue with many consumers.

- All message producers use the same inbound adapter, all message consumers (MDBs) the same outbound one.  In fact all front node are exactly the same (same AS, same config, same deployed artifacts), same for all server nodes.

Now here is my problem :

the application is a multi-tenant one, for a given queue, some tasks (message processing) must not be processed in parallel for a given tenant, we set up so [message grouping][1] to handle this constraint. The message group is the tenant name and is set by the message producer when sending the message :

    message.setStringProperty("JMSXGroupID", tenantName);

On a platform we have 1000+ tenant (so 1000+ different message groups) and for a given queue 3 consumers per server and 3 tasks servers.

When monitoring this queue on the messaging server we can so see thousands messages in queue and 9 consumers. We expects message to be delivered 9 by 9 but in practice the in-delivery message count never exceed 1.

What's the problem here ? is the topology suitable for our needs ?

  [1]: http://docs.jboss.org/hornetq/2.2.14.Final/user-manual/en/html_single/#message-grouping

[jbertram]

As I'm sure you're aware, a queue has first-in-first-out (i.e. FIFO) semantics.  Grouped messages basically act like a bottleneck on the queue since they ensure serial message processing.  Here's a simple example...

Consider groups A, B, and C which contain two messages each for a total of 6 messages in the queue.  Also consider that they are in the order A, A, B, B, C, C.  Now consider 3 different consumers each consuming a different group.  The consumer for group A will receive the first message, process it, and acknowledge it so that it is removed from the queue.  Then it will receive the second message, process it, and acknowledge it so that it is removed from the queue.  During the time the consumer for group A is busy with these 2 messages no other messages can be consumed from the queue.  Only when the second message is acknowledged can the consumer for group B actually receive the first message in group B.  Once the consumer for group B acknowledges both of its messages the consumer for group C can finally receive the messages in its group.  This behavior is governed by the FIFO semantics of the queue.  The messages in group B can't leap-frog the messages in group A and be consumed before all the messages in group A are consumed.  The same goes for the messages in group C.

Since all your messages are in a group I would never expect the in-delivery count to exceed 1.

[gab-_-]

Thanks a lot for your quick, very clear response ! I was afraid about something like this. Is there a way in hornetq (in preference using JMS 1.1 API) to setup the behavior we expect ? (Message must also be redelivered in case of processing failure).

FYI the question has been duplicated on SO (java - JMS topology (Queue with multiple consumers) and message grouping - Stack Overflow). I'd be pleased to accept your answer there if you have an account. Otherwise Can i copy you answer with a reference here ?

[jbertram]

Thanks a lot for your quick, very clear response ! I was afraid about something like this. Is there a way in hornetq (in preference using JMS 1.1 API) to setup the behavior we expect ? (Message must also be redelivered in case of processing failure).

I'm not clear on what specific behavior you expect.  Could you outline your requirements?

FYI the question has been duplicated on SO (java - JMS topology (Queue with multiple consumers) and message grouping - Stack Overflow). I'd be pleased to accept your answer there if you have an account. Otherwise Can i copy you answer with a reference here ?

I don't have an SO account, but feel free to copy my answer if you like.

[gab-_-]

I would like to be able to guarantee that, as specified in your documentation, "Messages in a message group are always consumed by the same consumer" but getting rid of the FIFO blocking behavior on message retrieval (for which i can't find any reference in your documentation neither in JMS 1.1 API). In other words I'd like each idle consumer to retrieve the first message in the queue matching its groups.

This said,I'am open to any alternative but I would like to avoid to set up a different configuration for my consumers (avoiding so a SPOF on a specific consumer, any consumer being able to be elected in place of a crashed one). In the described architecture it means that all tasks servers are still perfectly identical (same config, same deployed artifacts).

[jbertram]

...but getting rid of the FIFO blocking behavior on message retrieval (for which i can't find any reference in your documentation neither in JMS 1.1 API).

You can't simply get rid of the FIFO semantics as that would change the queue into some other kind of data structure.  I guess you don't find an explanation of this either in the HornetQ documentation or in the JMS API is because it is assumed, kind of like saying the sky is blue.

In other words I'd like each idle consumer to retrieve the first message in the queue matching its groups.

If that's the case then I think you would need a specific queue for each group.  It's theoretically possible that you could implement a remoting interceptor to catch the incoming messages, inspect their group, and redirect them to their own queue.  Just an idea.

Other ways to mitigate the consumption delay:

• Make the consumers faster (e.g. ack the messages right when you get them instead after they've finished processing).
• Make the group sizes smaller (e.g. instead of sending lots of messages in a single group perhaps batch them together into one large message if possible).

At the end of the day you're trading performance for guaranteed grouping (which requires serial consumption).  If you don't like the performance trade-off then you might want to re-evaluate your need for grouped messages.