HornetQ Subscriber Limit? Use DB Instead?

[unsavory]

We are currently using HornetQ 2.1.0 GA as our JMS provider on a couple of our production application servers.  We absolutely love it btw, and have been using it to process approximately 25 messages per second with absolutely no problems.

We would like to use it for another use case in our environment and are curious if HornetQ was designed to handle something like this.  Specifically, we would like to produce messages to a non-durable last value queue with somewhere around 10 producers.  And then we need to be able to consume these messages with potentially hundreds of concurrent subscribers using a message selector to only pull the messages out by subscriber id.  The messages would be about the size of say a map of a simple user profile's information (id, name, address, phone, etc, etc).

What makes this difficult is that these consumer threads will be initiated via web request.  So I was thinking we would pool a group of connected consumers, and dynamically switch out the message selector for the subscriber on the pooled object when it was checked out.

My questions are:

A.  Can you dynamically switch out a message selector on an open consumer so that we can only fetch messages for the thread's end user?

B.  Can HornetQ handle having a couple hundred concurrent consumers on a single queue (consumers will run on same appserver as hornetq)?

Note:  The queue would have probably a couple thousand entries at any given time as we are only interested in the latest information.

I guess I don't know what the overhead would be in HornetQ when using message selectors with a large volume of consumers.  I'm unsure if HornetQ can handle publishing and subscribing with this amount of volume while using selectors.  Is this something that would be more suited to a database table?

Thanks for any guidance you may be able to provide.

[timfox]

I think the most efficient way to do this would be to use a topic, not a queue, and create and cache a non durable subscriber per subscriber id, when creating the subscription specify the subscriber id in a selector.

Should be no problem to cache hundreds if not thousands of subscribers. If you put the subscribers in a map keyed with subscriber id you can easily look them up when the web request comes in.

[unsavory]

Thank you very much for the reply, Tim.  I like your idea a lot.

The only question I have, is how to handle the last value piece using a topic.  Can a last value topic be created?  Or will I need to manage the messages in the topic manually (ie: programatically make sure only 1 message per subscriber is on the topic)?

The thing is, we only want our subscribers to have a maximum of 1 message to retrieve from the topic.  And that 1 message should always be the last published message for that subscriber.  I just don't see any examples of a last value topic, like there is for a queue.

[timfox]

A JMS topic is just a set of subscriptions.

Each subscription is just a queue, like any other queue so the same semantics apply.

[unsavory]

Thanks Tim!

We are up and running.  The problem space was a little more complicated than I originally thought because we also need to know when to produce messages based on whether or not a client has been active in the past xx minutes, and because requests come in frequently, we did not want to fetch more data than is absolutely necessary.

Here is what we ended up doing in case it might help someone else.

1.  A web request comes in and we check for any messages on the topic using a topic selector.  We cache these topic consumers in a concurrent hashmap as suggested.  If no topic consumer exists for this client, we create a new one and place it into the map.  After checking for messages and consuming them if present, we ping a "client registry" (another concurrent hashmap) with a last request timestamp so we can know if the client is active.

2.  There is a separate command queue that is used for retrieving data for the clients.  The command queue producers know how often to produce commands of various types and only produce at a given rate (ie: every 5 minutes per client for this command, every 10 minutes per client for that command).

3.  There are producers that check the entire client registry every xx seconds and if the timestamps are not expired, they use the command producers described in step 2 to produce data retrieval commands.  Expired registry entries are removed from the map.

4.  There is xx number of command consumers that feed off the command queue fetching data, and then finally place messages back onto the client's topic containing said data.

All this is a very complicated and round about way of saving database resources so we do not have to fetch data for our web clients except at given intervals for different types of data.

Now, I do have another unrelated question regarding system resources with all these cached topic consumers which I will post in another thread.

Thanks again for your help!