4 Replies Latest reply on May 20, 2011 5:09 PM by Clebert Suconic

    messages in journal, journal gets big

    Armin Haaf Newbie

      Hi

       

      we had an OutOfMemoryError in and are currently investigating the problem

       

      10:38:52,521 ERROR [STDERR] java.lang.OutOfMemoryError: Java heap space

      10:38:52,524 ERROR [STDERR]     at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)

      10:38:52,524 ERROR [STDERR]     at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)

      10:38:52,525 ERROR [STDERR]     at org.hornetq.core.journal.impl.NIOSequentialFile.fill(NIOSequentialFile.java:112)

      10:38:52,525 ERROR [STDERR]     at org.hornetq.core.journal.impl.JournalFilesRepository.createFile(JournalFilesRepository.java:473)

      10:38:52,525 ERROR [STDERR]     at org.hornetq.core.journal.impl.JournalFilesRepository.takeFile(JournalFilesRepository.java:419)

      10:38:52,528 ERROR [STDERR]     at org.hornetq.core.journal.impl.JournalFilesRepository.pushOpenedFile(JournalFilesRepository.java:386)

      10:38:52,530 ERROR [STDERR]     at org.hornetq.core.journal.impl.JournalFilesRepository$1.run(JournalFilesRepository.java:342)

      10:38:52,531 ERROR [STDERR]     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

      10:38:52,531 ERROR [STDERR]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

      10:38:52,532 ERROR [STDERR]     at java.lang.Thread.run(Thread.java:662)

       

      maybe this is related to a big journal of about 800MB which we cannot explain, as we do not see any pending messages in any of the queues/topics

       

      However, we used org.hornetq.core.persistence.impl.journal.PrintData to show the content of the journal and can see about 100000 messages in the journal. If I interpret the data correct, the messages are for (2) topic subscriptions, which are no longer subscribed, resp. we cannot see any subscription via the jmx management. As we cannot see any subscriptions, we cannot delete the messages.

       

      A solution is to restart hornetq and drop the journal. But you know, we will loose all other messages also (we can now live with the loss, but later on this is a problem).

       

      So are there more sophisticated methods

      - to check if there are pending messages on the subscription

      - the cleanup the journal resp. to remove the messages

       

       

      we are using hornet 2.2.2 with jboss 5.1. Only hornetq is running on jboss...

       

       

      Thanx

       

      Armin

        • 1. messages in journal, journal gets big
          Clebert Suconic Master

          Don't you see the Queues on the bindings when using PrintData?

           

           

          PrintData will be divided into two steps. I - will print the adds and deletes. II - Will print only surviving records.

           

          I'm always interested in broken data to avoid any issues for other users. If you send it to me I could take a look.

           

          (records are just alleatory data for me, but I wouldn't get it if that's something that would create me lliability).

          • 2. messages in journal, journal gets big
            Armin Haaf Newbie

            In fact, i do not fully understand the journal data.  But I interpreted the surving records as messages that are not yet delivered, i hope I was correct.

             

            I do not know if the printdata is incorrect, or I'm not able to see the messages anywhere in the jmx console. But I think the PrintData is correct, as we have a big heap allocation after startup of jboss and the journal has always the same footprint in the filesystem.

             

            Which data do you need? The whole hornetq data directory (about 1GB unzipped) or the result of the PrintData (6MB zipped)

            • 3. messages in journal, journal gets big
              Clebert Suconic Master

              The whole data... you will have to send me an email probably

               

               

              You will probably have to place it on some http server (I doubt it will fit an email even zipped)

               

              csuconic

               

              at

               

               

              redhat

              • 4. messages in journal, journal gets big
                Clebert Suconic Master

                Maybe you deleted the JMS binding but not the core-queue.

                 

                 

                I - make sure you delete the core queues associated with your addresses.

                II - also configure paging or blocking so you won't get to a scenario where you have more data than it fits on the JVM.

                III - process your data with a bit of more memory, then limit your data later.

                 

                 

                (You sent me your data in pvt and I didn't see anything damaged).