4 Replies Latest reply on Jan 26, 2010 3:59 AM by timfox

    Restarting HornetQ after paging starts

    johnnysoccer

      We have seen in several instances where we are unable to restart HornetQ after reaching certain memory thresholds.

       

      Scenario 1)  HornetQ has crashed due to running out of memory either due to misconfiguration of the paging parameter(s) or unexpected resource consumption as a combination of all the processes running on the machine.

       

      Scenario 2) We have reached a point where we have entered paging, and there are a large number of paging files (40+) meaning we have been paging for a while, typically do to one of our durable subscribers not being available to consume messages.

       

      In both cases, is now in a stopped state, and we attempt to start it back up again.  HornetQ never manages to start up, it either completely hangs, or throws an out of memory error.  In both instances, the only way to get HornetQ to restart was to bump up the -Mmx parameter in the start script to significantly higher than it was (roughly 50% higher) in order to get HornetQ to start back up again. (or clean out the journal and paging directories, which would not be an option for a system in production).

       

      Our typical configuration starts HornetQ with an -Xmx1024m and a paging configuration as follows:

       

            <address-setting match="jms.topic.Replication.#">

               <max-size-bytes>524288000</max-size-bytes>

               <page-size-bytes>10485760</page-size-bytes>

               <address-full-policy>PAGE</address-full-policy>

            </address-setting>     

      Where there are two topics that would match the address setting (we have assumed that the max-size-bytes is a total for all the queues identified by the match parameter). We have noticed, through jconsole, that despite having the max-size-bytes set at 1/2 Gig, that memory typically will grow to 900 Mb before paging starts, which also seems wrong, but that is what we have observed.
      Finally, all of our subscribers are durable, and in our test cases, there have typically been 4-5 subscribers, though we expect to have more than 400 in a normal situation.
      Is seems that based on this configuration, there should always be enough memory for HornetQ to restart, but this has happened many times (2.0.0.GA)
      Any suggestions on settings we should be using to protect the system better on startup.
        • 1. Re: Restarting HornetQ after paging starts
          timfox

          I will check, but I believe the max-size-bytes is per matching address, *not* in total.

           

          That would explain why you're running out of memory. Try halving the value and see if the problem still occurs.

          • 2. Re: Restarting HornetQ after paging starts
            johnnysoccer

            I will change the values to be per queue within the match filter. I can see how this will definitely help with our running memory issues, but will this fix the problem of HornetQ starting back up. For example, say we have a memory problem and realize we need to reduce the size of max-size-bytes to a smaller value, we change our configuration, on startup, will HornetQ recognize this change and try to load less of the persisted messages (those in the journal folder) into memory and generate additional paging files?

             

            This is the behavior I would expect, but I'm not certain it is what we have seen.

             

            thanks for the response,

             

            John

            • 3. Re: Restarting HornetQ after paging starts
              clebert.suconic

              The max-size is per matching address.

               

              We used to have a global size for paging but we decided removing it, as it was a little messy. (Things like what address should we start depaging when memory is freed. You would eventually get the sensations of being locked as other address is holding up the memory.. etc)

              • 4. Re: Restarting HornetQ after paging starts
                timfox

                No, you must consume all the messages in memory first, before consuming the messages from paging.