10 Replies Latest reply on Jul 4, 2011 9:37 AM by ronnys

    HornetQ 2.1.2 paging issue under constant load

    ronnys

      Hi Clebert,

       

      I've a question regarding the HornetQ 2.1.2 paging:

       

      I did some performance tests with a simple JMS client today. The client is just having a producer and a consumer thread. The producer thread simply constantly produces 1Kb ByteMessages in a transacted session into a queue; each message gets committed separately. The consumer simply consumes all produced messages in another transacted session, each message gets committed separately as well; it is started later than the producer to enforce paging. Code is attached.

       

      The address of the used queue is configured like this:

       

            <address-setting match="jms.queue.divtest_main">

             <max-delivery-attempts>-1</max-delivery-attempts>

             <max-size-bytes>33554432</max-size-bytes>

             <page-size-bytes>16777216</page-size-bytes>

             <address-full-policy>PAGE</address-full-policy>

            </address-setting>      

       

      Now, the producer runs, the queue gets filled up and finally paging starts at ~ 13000 messages in the queue. So far so good. As soon as the consumer kicks in, the queue length decreases, the message count shown via JMX jumps a bit up and down as soon the messages get depaged and finally ends up around 0. The new paging files written to disk get smaller and smaller before they get processed and deleted. However, more than 10000 files are being written in a row, most of them about 2Kb big. Paging doesn't stop for *minutes* after the queue is reported as more or less empty. HornetQ still pages each message out to disk just to depage it more or less immediately. This impacts performance heavily. Paging only gets switched off if the sender gets stopped for some seconds or if the queue probably gets empty for a short time due to some other event. The log (with enabled trace logs for org.hornetq.core.paging.impl.PagingStoreImpl) is attached. The paging and depaging seems to be a bit disconnected here, paging could stop much earlier in my point of view.

       

      Is this issue somehow addressed in the new HornetQ 2.2 paging?

       

      Best regards,

      Ronny

        • 1. HornetQ 2.1.2 paging issue under constant load
          clebert.suconic

          The old model would depage messages as needed.

           

          We can't stop paging mode until all the messages are consumed, otherwise you would have ordering issues.

           

           

          At the end of paging.. when you depage.. you need to open a new page, and then imediately another message is paged...   That means you will be depaging and paging quickly (depending on your use case).

           

           

          Also, the old model didn't have sync optimized... which is now implemented.

           

           

          The newer model would still let you use the current page, so this shouldn't be an issue.

          • 2. HornetQ 2.1.2 paging issue under constant load
            ronnys

            Hi Clebert,

             

            thanks a lot for the reply!

             

            Best regards,

            Ronny

            • 3. Re: HornetQ 2.1.2 paging issue under constant load
              ronnys

              Hi Clebert,

               

              hope you're doing well.

               

              We just did some performance tests with HornetQ 2.2.5 and we still see this issue. It is considered as a showstopper by our production support team. The throughput goes down to about 20% as soon as the paging starts - this is not nice but might be acceptable. However, we've a more or less constant flow of messages, the likelihood to have an empty queue - in order to stop the paging - for a short point in time is pretty low. Having HornetQ in a state, from which it cannot recover by its own is simply bad.

               

              There will be quite some producers connected to the queue. We can't shut them down all together, pausing them is a workaround, at least for some kinds of producers, but requires manual intervention

               

              Would it be possible to implement some mechanism into HornetQ, that measures the performance of the consumers and the queue length and stops or slows down producers as soon as it reaches a certain threshold? If it, for example, detects, that the current consumers could really empty the queue within 5s if there are no new messages, it could pause the producers, wait until the queue is empty (up to a certain second threshold to handle the case where consumers go/slow down) and then resume the producers to exit the paging mode.

               

              What do you think?

               

              Best regards,

              Ronny

              • 4. Re: HornetQ 2.1.2 paging issue under constant load
                clebert.suconic

                how you are consuming / producing?

                 

                 

                transactioned / non-tx / what ack mode / mdbs?

                 

                 

                 

                maybe you could speed up consumption by using different options.

                 

                 

                 

                anything we would do in this regard to slow down producers would have to be a management operation. And you are actually asking us to do a mix between blocking and paging. I'm not sure what we could do automatically.

                • 5. Re: HornetQ 2.1.2 paging issue under constant load
                  clebert.suconic

                  Is the initial test you posted still valid? You are committing on every message consumed and sent.

                   

                  Why don't you batch transactions if you really need transactions? you would be much faster by doing this.

                   

                   

                  commit on every 1000 messages (and the DB work you have in also 1000 messages). If you rollback just rollback the whole thing then and all the messages will be redelivered.

                   

                   

                  When you do Paging in Transactions, you will be updating a record on the journal as well.

                   

                   

                  Another option you have, would be to not send using transactions. You would avoid a lot of overhead. (By this I mean, only use transactions on consumption).

                  • 6. Re: HornetQ 2.1.2 paging issue under constant load
                    ronnys

                    Hi Clebert,

                     

                    thanks for your replies.

                     

                    Producer and consumer are transactional, the consumer is a JBoss-MDB. Every message is committed, using batched transactions or no transactions at all is unfortunately not possible here.

                     

                    Consumption is already at top-speed, it just takes seconds to process the backlog in the test. Afterwards, the queue length is always around 0 according to JMX, most of the time actually 0, but HornetQ does not leave the paging mode until we pause the producers for a second or in case we just hit the right condition coincidentally.

                     

                    The old test is still valid, however, an updated version is attached.

                     

                    I modified HornetQ (v.2.2.5 release version) a bit to use a timeslot when HornetQ switches to the next page file to pause the paging for a short time to resolve the race condition. Patch files are attached.

                     

                    It's not an elegant solution, more a hack, but it appears to work - at least during the tests. Don't know if has any side effects or is 100% correct at all, might be that it forces pauses under wrong circumstances as well. It still needs to wait for a page file switch, even if the queue is in the state described above. It would be nice if HornetQ could get a similar mechanism, but hopefully with a better way to detect when to delay #page() to get rid of the need to wait for the next page file switch to leave the page mode faster.

                     

                    HTH,

                    Best regards,

                    Ronny

                    • 7. Re: HornetQ 2.1.2 paging issue under constant load
                      clebert.suconic

                      You have high performance requirements...

                       

                      In your place I would send the messages using Persistent, non-transactionally. A message sent would be block on the producer what means you wouldn't leave a producer.send() until you had guarantees of delivery.

                       

                      That would speed up paging/depaging consumption by a good factory already.

                       

                      If you need transaction on depaging, I wouldn't use MDBs at all. I would use a regular consumer and batch the transaction. (say a block of 200 messages).

                       

                       

                      If you use these techniques you won't need any changes.

                      • 8. Re: HornetQ 2.1.2 paging issue under constant load
                        clebert.suconic

                        I'm looking at your patch. Do you mind to produce a "svn diff" instead for me?

                        • 9. Re: HornetQ 2.1.2 paging issue under constant load
                          ronnys

                          Hi Clebert,

                           

                          I agree with you regarding the performance improvements, however, this is something that cannot be changed right now. However, it might need to and I guess it will be changed in the future. On the other hand, as said, the consumer, where we see the issue, can easily cope with the amount of messages produced by the producers.

                           

                          Have you tried to run the test class to see the behaviour? Might be that we're just talking at cross-purposes. As soon as the consumer kicks in, it empties the queue within a pretty short time. Afterwards, with an empty queue, HornetQ is still paging and consumer and producer performance is equal but low for minutes. This is what concerns me.

                           

                          svn diff is attached.

                           

                          Best regards,

                          Ronny

                          • 10. Re: HornetQ 2.1.2 paging issue under constant load
                            ronnys