Could we please ask some advice?
Our setup is that we have a central HornetQ server, and a number of customer sites with HornetQ JMS bridges connecting to the server across the Internet. Each site consumes messages from a JMS topic.
All messages have the same priority, and all are roughly the same size. Message order is important. We are using Hornetq-2.1.2.Final.
Each customer site is in a geographically different area, and has a different quality of Internet connection. Quite often, a site will fail to both connect to and consume messages from the topic for a variable amount of time. Unfortunately, this causes the subscription's core queue to build up messages for the site, which very quickly fills the available memory for the address. This causes the address to start paging, and no other sites are able to pick up.
We know this is known behaviour; HORNETQ-325 may improve this, but we can't wait for the sceduled version. Here is the workaround we have designed (below). We have also seen this forum thread, and are worried about the number of copies of messages that would be generated if we had one divert per core queue (JMS topic subscription). We also want to avoid having to add diverts to the XML file for each site, as the number of sites can vary.
This largely assumes that if you have a number of paged messages on an address, and create a divert on the address after the messages are already in the page file, that any matching depaged messages will be diverted (as paging happens before message delivery). Is this a correct assumption, as we don't see this explicitly in the documentation?
Firstly, we will create an MBean managing a thread (or runnable) that will run periodically inside the HornetQ stand-alone server. The thread will perform this algorithm:
if address (for the topic) is not paging
do nothing (ie wait to restart loop)
find the core queue with the highest messageCount (number of messages in memory) for the address. Call this Q
Create a new durable queue ( lets call this H for "holding queue") whose name is based on the message selector (which are all of a similar format).
Create a divert from the topic to H, with the same message selector as Q, and a transformer to set the message priority lower to handle new messages
Move all existing messages from Q to H using the JMX interface (or would core API be better) with their original priority to ensure the order is preserved.
(This will free up memory in the address to allow HornetQ to re-evaluate the paging file, and divert any message from the paging file).
Wait a bit for the paging file to be read, then done (ie wait to restart loop).
At startup, the MBean will also inspect the "holding queues" that are created, and re-create any required diverts.
We're not sure yet exactly how we will deliver messages in "holding queues" to sites; we could either have the site use a JMS Bridge to pull messages from the queue (since you can't pull messages with a core bridge, and there would be firewall issues pushing a message with a core bridge) and then poll for the queue being empty to remove it, or we could do something to move the messages back into the queue and remove the divert (we would need to reset the message priority to keep the order of messages, and be careful of new messages arriving between moving the messages and removing the divert).
We know that most of our customer sites are fairly reliable at picking up messages but outages can occur and some of our customer sites are always temperamental so we are hoping that this approach might yeild a flexible approach with a minimum of impact on the bulk of our customers and without creating a XML configuration hell of diverts and bridges.
Is this a sensible approach, and does a better workaround exist?