4 Replies Latest reply on Mar 19, 2013 7:48 AM by yarek2013

    Paging folder is not shrinking

    yarek2013 Newbie

      Hello,

       

      I am overseeing production environment (as administrator with no insight into source code) with hornetq as messaging system (2.2.13) and noticed that for some time paging folder is always growing and never shrinking. Consumer side of the system was put offline to ensure all messages are processed by producers, but when all the queues became empty (checked with jboss-cli) paging folder didn't shrink a single bit. I wonder whether something is wrong with configuration or producers code (to which I have no access, but can only complain). Can you please advise where to look? The only sane explanation I was able to find is that messages in some queues are not acknowledged, could this be the case?

       

      Configuration looks as follows (messaging system):

       


                                      <persistence-enabled>true</persistence-enabled>
                                      <journal-file-size>10240000</journal-file-size>
                                      <journal-min-files>20</journal-min-files>
                                      <security-enabled>false</security-enabled>
                                      <transaction-timeout>2400000</transaction-timeout>
      
      
                                      
                                      <address-setting match="#">
                                                      <dead-letter-address>jms.queue.DLQ</dead-letter-address>
                                                      <expiry-address>jms.queue.ExpiryQueue</expiry-address>
                                                      <redelivery-delay>1</redelivery-delay>
                                                      <max-delivery-attempts>10</max-delivery-attempts>
                                                      <max-size-bytes>848576000</max-size-bytes>
                                                      <page-size-bytes>8485760</page-size-bytes>
                                                      <page-max-cache-size>10</page-max-cache-size>
                                                      <address-full-policy>PAGE</address-full-policy>
                                                      <message-counter-history-day-limit>10
                                                      </message-counter-history-day-limit>
                                      </address-setting>
      
      
      

        • 1. Re: Paging folder is not shrinking
          Yong Hao Gao Master

          I don't think using jboss-cli can read paged messages in your version of hornetq. It seems to me your messages are not consumed at all because you closed all consumers. If messages are not consumed and acked, the messages won't be deleted from page store.

          Can you try the latest version?

           

           

          Howard

          • 2. Re: Paging folder is not shrinking
            yarek2013 Newbie

            I am not sure whether consumers are consuming messages correctly, but say Friday evening all producers were stopped and only consumers were active. At this point queues had around 300 000 unprocessed/undelivered messages, which have been handled by Saturday evening. All these messages trigger some operation which is reflected by updating records in database. No operations happened on Sunday/Monday night and paging folder didn't shrink, consumers were up and running from Friday till Monday morning.

             

            We also tried enhancing consumer capabilities by 2 (in terms of CPU/RAM), network and database are not bottlenecks as well. Now queues have much less messages in RAM, but paging folder still grows and never shrinks. Even with less than 1 % of active end-users at night consumers just stay idle and paging never shrinks.

             

            I suspect that either something is wrong with configuration (can't see anything obvious) or something is not ok with the code.

             

            Unfortunately I can't update freely since it's in production, but may be this pattern could mean something?

             

            We are now growing our disk to buy time, but it can't work forever. Could it be that messages in some queues are not acked or something else could be going on?

            • 3. Re: Paging folder is not shrinking
              Yong Hao Gao Master

              The configuration you posted looks fine to me. It seems that either your consumers for some reason stopped consuming messages, or the server for some reason stopped delivering messages.

               

              I have some questions:

               

              Does the issue happen always or occassionally? Did you enlist the message receiving and DB update in a transaction? When it happens, do you get some error messages at any of your consumers or the server?

              Did you find that you have a lot of messages that haven't been received after Saturday ?

               

              Of course it's possible you have hit some paging bugs. But your use-case from HornetQ angle is not very uncommon. So I would suggest you set up a simple test where you first send many messages so that the paging happens, and then start  a consumer to consume those messages. When all messages are consumed, check the page directory and see if it shrinks. If it doesn't shrink in this case, do it again with a latest version of HornetQ so we can be clearer what could be the reason.

               

              Howard

              • 4. Re: Paging folder is not shrinking
                yarek2013 Newbie

                Does the issue happen always or occassionally?

                This issue happens always, it's just recently that disk space became a pressing issue

                 

                 

                Did you enlist the message receiving and DB update in a transaction?

                 

                I believe so, when producer receives a messages it's ID is inserted in the database for tracking purposes and sent to jms. Producers and consumers update message status in the database.

                 

                When it happens, do you get some error messages at any of your consumers or the server?

                 

                I didn't spot any dependency between getting errors and growing paging folder. Though sometimes errors happen and presumably messages go back to originating queue.

                 

                 

                Did you find that you have a lot of messages that haven't been received after Saturday ?

                 

                I assume no, but can't tell for sure. Generally if there is a problem end-users complain more than usual.

                 

                 

                Of course it's possible you have hit some paging bugs. But your use-case from HornetQ angle is not very uncommon. So I would suggest you set up a simple test where you first send many messages so that the paging happens, and then start  a consumer to consume those messages. When all messages are consumed, check the page directory and see if it shrinks. If it doesn't shrink in this case, do it again with a latest version of HornetQ so we can be clearer what could be the reason.

                 

                In the coming days I will set up test environment for custom services, perform some tests according to your suggestion and post here the results.

                 

                Thanks for advice.