2 Replies Latest reply on Sep 10, 2010 7:55 AM by robertjlee

    Avoiding blocking consumption from a topic due to paging

    robertjlee

      Could we please ask some advice?

       

      Our setup is that we have a central HornetQ server, and a number of customer sites with HornetQ JMS bridges connecting to the server across the Internet. Each site consumes messages from a JMS topic.

       

      All messages have the same priority, and all are roughly the same size. Message order is important. We are using Hornetq-2.1.2.Final.

       

      Each customer site is in a geographically different area, and has a different quality of Internet connection. Quite often, a site will fail to both connect to and consume messages from the topic for a variable amount of time. Unfortunately, this causes the subscription's core queue to build up messages for the site, which very quickly fills the available memory for the address. This causes the address to start paging, and no other sites are able to pick up.

       

      We know this is known behaviour; HORNETQ-325 may improve this, but we can't wait for the sceduled version. Here is the workaround we have designed (below). We have also seen this forum thread, and are worried about the number of copies of messages that would be generated if we had one divert per core queue (JMS topic subscription). We also want to avoid having to add diverts to the XML file for each site, as the number of sites can vary.

       

       

      Proposed workaround:

       

      This largely assumes that if you have a number of paged messages on an address, and create a divert on the address after the messages are already in the page file, that any matching depaged messages will be diverted (as paging happens before message delivery). Is this a correct assumption, as we don't see this explicitly in the documentation?


      Firstly, we will create an MBean managing a thread (or runnable) that will run periodically inside the HornetQ stand-alone server. The thread will perform this algorithm:

       

      LOOP:
      if address (for the topic) is not paging
      then,
        do nothing (ie wait to restart loop)
      otherwise,
        find the core queue with the highest messageCount (number of messages in memory) for the address. Call this Q
        Create a new durable queue ( lets call this H for "holding queue") whose name is based on the message selector (which are all of a similar format).
        Create a divert from the topic to H, with the same message selector as Q, and a transformer to set the message priority lower to handle new messages
        Move all existing messages from Q to H using the JMX interface (or would core API be better) with their original priority to ensure the order is preserved.
        (This will free up memory in the address to allow HornetQ to re-evaluate the paging file, and divert any message from the paging file).
        Wait a bit for the paging file to be read, then done (ie wait to restart loop).

       

      At startup, the MBean will also inspect the "holding queues" that are created, and re-create any required diverts.

       

      We're not sure yet exactly how we will deliver messages in "holding queues" to sites; we could either have the site use a JMS Bridge to pull messages from the queue (since you can't pull messages with a core bridge, and there would be firewall issues pushing a message with a core bridge) and then poll for the queue being empty to remove it, or we could do something to move the messages back into the queue and remove the divert (we would need to reset the message priority to keep the order of messages, and be careful of new messages arriving between moving the messages and removing the divert).

       

      We know that most of our customer sites are fairly reliable at picking up messages but outages can occur and some of our customer sites are always temperamental so we are hoping that this approach might yeild a flexible approach with a minimum of impact on the bulk of our customers and without creating a XML configuration hell of diverts and bridges.

       

      Is this a sensible approach, and does a better workaround exist?

        • 1. Re: Avoiding blocking consumption from a topic due to paging
          clebert.suconic

          I think it would be a very reasonable way to make it work.

           

          Actually, you just gave me a few ideas on how to address the issue.

           

          We have been thinking on a way to implement flow-to-disk on slow consumers what would release the whole topic. Having an auxiliary queue for slow consumers would be a reasonable way.  Of course it's just a thought now.. I would need to put more thought on it (for the permanent solution on 2.2)

          1 of 1 people found this helpful
          • 2. Re: Avoiding blocking consumption from a topic due to paging
            robertjlee

            thank you; we were worried that the solution was over-complicated compared to the problem.

             

            There is only one snag that we've noticed so far: it only works in the case where every message goes to exactly one consumer, which sadly isn't the case in our system.

             

            We have a couple of ideas on how to get around that, the simplest of which seems to be to have an intermediary Queue that is consumed by non-exclusive diverts to put messages into the topic but one of the diverts modifies the message's headers to prevent it from going into the wrong topic subscription (using the subscription's existing message selector), the other non-exclusive diverts put the messages into holding queues.

             

            Unfortunately, this is no longer a general solution to the problem, as it requires suitable message selectors and known header formats, and it will now create at least one unnecessary copy of every message coming in, even if it isn't diverted to a holding queue.