3 Replies Latest reply on Dec 8, 2010 11:33 AM by robertjlee

    High memory requirements on startup

    robertjlee

      We had a problem yesterday where we had to restart the HornetQ server after an OutOfMemory error.

       

      We shut down the server and compacted the journal, which left around 3GiB of data in the journal directory (we kept a copy of the journal, which is around 300MiB compressed).

       

      We get an OutOfMemoryError (java Heap Space) on HornetQ startup. The JVM is limited to 16GiB (-Xmx setting), but the only address are a JMS topic (max-size-bytes=4294967296), and the DLQ, Expiry Queue, and one JMS queue (max-size-bytes=1048576), giving a total max size across all addresses of 4.1GiB.

       

      I know that the journal can contain deletes that would not occupy space in RAM, but I would have expected those to be removed during journal compaction. Is the additionally required space to be expected, or could this be a bug?

        • 1. Re: High memory requirements on startup
          robertjlee

          I've tried to open the same data using the latest build of TRUNK, and it still gives an OutOfMemory error, using a limit of 10GiB of heap space (the test machine I have access to doesn't have 16GiB of RAM so I can't do an exact like-for-like test). Can this be correct behaviour when there shouldn't be more than 4.1GiB of messages in RAM at any point?

          • 2. Re: High memory requirements on startup
            clebert.suconic

            I'm not sure without seeing a sample of the data.

             

            Any way you could share your data with us? (We never look at the actuall body messages.. they are all just random bytes for us).

             

            (however I wouldn't like to get any data that will create liability on me even though I wouldn't look at the data, say if you are sending passwords or other financial sensitive data. Maybe you could the export tool to mask the data if that's the case (after some editing)).

             

             

            BTW: The ExportTool may come handy to you on analyze the data.

            • 3. Re: High memory requirements on startup
              robertjlee

              Hi Clebert,

               

              Sorry for the delay. There's no sensitive data in the journal, and we've archived the entire data directory:

               

              http://dev.actual-systems.com/hornet/data-574452.tar.gz