6 Replies Latest reply on Sep 17, 2012 2:49 PM by jbertram

    Why does 'blocking' and 'paging' have to be mutually exclusive in regards to JMS Queues?

    garneke11

      This has got to be one of my top wishes for HornetQ enhancements.

       

      I would like to set a blocking limit for a Queue, but I do not necessarily want all of the messages to be forced to remain in memory.

       

      In my application I handle thousands of (large) messages and typically in surges.  I need HornetQ to act as a queue and manage the throughput.

      I may have 10 Queues running and I do not have enough heap space to set the blocking limit to the size I would like. 

      Furthermore, I can not rely on having enough disk space to page everything. If one application goes down things can backup quickly.

       

      I have configured my Queues to be persistent because I can not lose any messages - so any speed advantages by keeping everything in memory is probably moot. 

       

      The system I work with involves a lot of desperate applications and Blocking allows the destination Queue to force the system to backup and keeps the flow of data contained at the source until it can be handled.  In theory this is great, however, there is performance issues when blocking that can be managed most efficiently by controlling the limit at which the queue blocks.  This optimal limit can not be used because in worst case scenarios all 10 Queues would require too much memory for the JVM.  I have exceeded those boundaries before, and things get squirrelly, then HornetQ cannot restart and I end up throwing data away.

       

      To me this does not seem to be an unreasonable configuration - blocking and paging working together.  Am I missing something obvious, or would this just be harder to implement than it sounds.

        • 1. Re: Why does 'blocking' and 'paging' have to be mutually exclusive in regards to JMS Queues?
          garneke11

          This bit me on the ass again...  Is there no one that can explain why 'blocking' and 'paging' have to be mutually exclusive?

           

          Why can't I ( for example ) set up 15 queues to page at 10Mb and block when a queue has accumulated 250Mb of pending messages?

          The heap memory requirements would be minimal and the disk space would be managable.  This would solve so many problems from me.

           

          Come on Tim Fox help me out. You are good at pointing out the obvious to idiots like me.  What am I missing?  Why is this a bad idea?

          • 2. Re: Why does 'blocking' and 'paging' have to be mutually exclusive in regards to JMS Queues?
            clebert.suconic

            We only have page / block on 2.2, and page/block/fail on the next 2.3.Beta

             

             

            You could create a JIRA for failure under paging... that would be a feature request. (I don't think I would do block and paging, I would do fail and paging... throwing a message like user has achieved maximum size on the page.. or something like that).

             

            can you create the JIRA?

             

             

             

            BTW: Tim Fox is not working with us any longer, I'm the project Lead and the team is composed by Justin Bertram, Francisco Borges, Andy Taylor, Jeff Mesnil and Howard Gao.

             

            "You are good at pointing out the obvious to idiots like me. "

             

            And I'm not as good as Tim was on that.. I try to be nice to everybody. (... *try*... )

             

            on this case your question was a very good question that it even makes sense to open a feature request... I don't think we can do it before 2.3.0.GA though... otherwise it will never get out. (it will be probably 2.3.1)

            • 3. Re: Why does 'blocking' and 'paging' have to be mutually exclusive in regards to JMS Queues?
              garneke11

              Thanks Clebert.

               

              I appreciate your response. 

               

              My application needs to run 24x7 and needs to run headless.  For that reason, I can not stop the application for anything other than critical errors and filling up the queue would not be considered critical.  It would be just a normal weekend surge.  Blocking allows the application to continue to run smoothly during these surges, except for those few instances where a transaction set causes too many messages to be queued before the commit is given and the queue blocks. 

               

              Chicken or egg senerio at that point.  Can not consume the messages because they have not been commited, but I can not commit them because they have not all been written. 

              The only solution is to increase the max-size-bytes or switch to paging to get through the block.  The problem is that paging alone will surely fill up my hard drive on a bad weekend, and increasing my max-size-bytes for all of my queues would exceed my jvm heap maximum.

               

              Block and page seems to be a great solution for me.

               

              I will attempt a jira ( feature request ).  

               

              * Tim was entertaining but he knew his stuff. *

               

              Thanks again for your help.

              • 4. Re: Why does 'blocking' and 'paging' have to be mutually exclusive in regards to JMS Queues?
                clebert.suconic

                My application needs to run 24x7 and needs to run headless.  For that reason, I can not stop the application for anything other than critical errors and filling up the queue would not be considered critical.  It would be just a normal weekend surge.  Blocking allows the application to continue to run smoothly during these surges, except for those few instances where a transaction set causes too many messages to be queued before the commit is given and the queue blocks.

                 

                 

                You could retry at your client.

                 

                If we do the feature it is probably easy to either block or fail. Most users so far have been complaining about blocking.

                 

                 

                * Tim was entertaining but he knew his stuff. *

                 

                 

                I also know a thing or two

                • 5. Re: Why does 'blocking' and 'paging' have to be mutually exclusive in regards to JMS Queues?
                  garneke11

                  Is there documentation on the FAIL option found in 2.3.beta anywhere? 

                  Does it allow for paging the memory as well? 

                  Is the fail interpretted as an exception on the client? 

                   

                  If so - this does sound like it could work. 

                  It would also allow for logging messages to record the client retries ( less mystery as to why the client stop writing ).

                   

                  Thanks

                  • 6. Re: Why does 'blocking' and 'paging' have to be mutually exclusive in regards to JMS Queues?
                    jbertram

                    Here's the commit.  You'll find the documentation update in there as well.