10 Replies Latest reply on Dec 29, 2010 10:53 AM by Clebert Suconic

    OutOfMemoryError on server start with "big" journal

    bodrin b. Newbie

      Hi,

       

      I have found some tests here in the forum and played with them. I have a durable queue with producer (and consumer, but not included here). So I have sent some messages and then and I got OutOfMemoryError when trying to start the server .. see below.

      I have turned on pagination and can't think of any other reason why this is happening, so I'm sending to you the test (two classes) and the bindings,journal and paging directories, because I have not found the exact way to reproduce it, but may be you can analyse the DB if you are interested

       

      Best regards,

      bodrin

       

      <quote>

      java.lang.OutOfMemoryError: Java heap space
      at java.nio.HeapByteBuffer.<init>(Unknown Source)
      at java.nio.ByteBuffer.allocate(Unknown Source)
      at org.hornetq.core.journal.impl.NIOSequentialFileFactory.newBuffer(NIOSequentialFileFactory.java:80)
      at org.hornetq.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:453)
      at org.hornetq.core.journal.impl.JournalImpl.load(JournalImpl.java:1760)
      at org.hornetq.core.journal.impl.JournalImpl.load(JournalImpl.java:1409)
      at org.hornetq.core.persistence.impl.journal.JournalStorageManager.loadMessageJournal(JournalStorageManager.java:777)
      at org.hornetq.core.server.impl.HornetQServerImpl.loadJournals(HornetQServerImpl.java:1220)
      at org.hornetq.core.server.impl.HornetQServerImpl.initialisePart2(HornetQServerImpl.java:1070)
      at org.hornetq.core.server.impl.HornetQServerImpl.start(HornetQServerImpl.java:313)
      at Test.<init>(Test.java:69)
      at Test.main(Test.java:232)
      java.lang.Exception: Java heap space
      at org.hornetq.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:764)
      at org.hornetq.core.journal.impl.JournalImpl.load(JournalImpl.java:1760)
      at org.hornetq.core.journal.impl.JournalImpl.load(JournalImpl.java:1409)
      at org.hornetq.core.persistence.impl.journal.JournalStorageManager.loadMessageJournal(JournalStorageManager.java:777)
      at org.hornetq.core.server.impl.HornetQServerImpl.loadJournals(HornetQServerImpl.java:1220)
      at org.hornetq.core.server.impl.HornetQServerImpl.initialisePart2(HornetQServerImpl.java:1070)
      at org.hornetq.core.server.impl.HornetQServerImpl.start(HornetQServerImpl.java:313)
      at Test.<init>(Test.java:69)
      at Test.main(Test.java:232)
      Caused by: java.lang.OutOfMemoryError: Java heap space
      at java.nio.HeapByteBuffer.<init>(Unknown Source)
      at java.nio.ByteBuffer.allocate(Unknown Source)
      at org.hornetq.core.journal.impl.NIOSequentialFileFactory.newBuffer(NIOSequentialFileFactory.java:80)
      at org.hornetq.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:453)
      ... 8 more

      java.lang.OutOfMemoryError: Java heap space

      at java.nio.HeapByteBuffer.<init>(Unknown Source)

      at java.nio.ByteBuffer.allocate(Unknown Source)

      at org.hornetq.core.journal.impl.NIOSequentialFileFactory.newBuffer(NIOSequentialFileFactory.java:80)

      at org.hornetq.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:453)

      at org.hornetq.core.journal.impl.JournalImpl.load(JournalImpl.java:1760)

      at org.hornetq.core.journal.impl.JournalImpl.load(JournalImpl.java:1409)

      at org.hornetq.core.persistence.impl.journal.JournalStorageManager.loadMessageJournal(JournalStorageManager.java:777)

      at org.hornetq.core.server.impl.HornetQServerImpl.loadJournals(HornetQServerImpl.java:1220)

      at org.hornetq.core.server.impl.HornetQServerImpl.initialisePart2(HornetQServerImpl.java:1070)

      at org.hornetq.core.server.impl.HornetQServerImpl.start(HornetQServerImpl.java:313)

      at Test.<init>(Test.java:69)

      at Test.main(Test.java:232)

      java.lang.Exception: Java heap space

      at org.hornetq.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:764)

      at org.hornetq.core.journal.impl.JournalImpl.load(JournalImpl.java:1760)

      at org.hornetq.core.journal.impl.JournalImpl.load(JournalImpl.java:1409)

      at org.hornetq.core.persistence.impl.journal.JournalStorageManager.loadMessageJournal(JournalStorageManager.java:777)

      at org.hornetq.core.server.impl.HornetQServerImpl.loadJournals(HornetQServerImpl.java:1220)

      at org.hornetq.core.server.impl.HornetQServerImpl.initialisePart2(HornetQServerImpl.java:1070)

      at org.hornetq.core.server.impl.HornetQServerImpl.start(HornetQServerImpl.java:313)

      at Test.<init>(Test.java:69)

      at Test.main(Test.java:232)

      Caused by: java.lang.OutOfMemoryError: Java heap space

      at java.nio.HeapByteBuffer.<init>(Unknown Source)

      at java.nio.ByteBuffer.allocate(Unknown Source)

      at org.hornetq.core.journal.impl.NIOSequentialFileFactory.newBuffer(NIOSequentialFileFactory.java:80)

      at org.hornetq.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:453)

      ... 8 more

      </quote>

        • 1. Re: OutOfMemoryError on server start with "big" journal
          Yong Hao Gao Master

          How is your hornetq configuration like? did you set max-size-bytes?

          • 2. Re: OutOfMemoryError on server start with "big" journal
            bodrin b. Newbie

            Hi, the test is into the zip attached above, but anyway here it is:

            Configuration configuration = new ConfigurationImpl();
            configuration.setPersistenceEnabled(true);
            configuration.setSecurityEnabled(false);
            configuration.setBindingsDirectory("D:/hq/bindings");
            configuration.setJournalDirectory("D:/hq/journal");
            configuration.setPagingDirectory("D:/hq/paging");
            configuration.setJournalType(JournalType.NIO);
            //configuration.set
            configuration.getAcceptorConfigurations()
            .add(
            new TransportConfiguration(InVMAcceptorFactory.class
            .getName()));
            server = HornetQServers.newHornetQServer(configuration);
            AddressSettings settings = new AddressSettings();
            settings.setDeadLetterAddress(new SimpleString(DLQ_QUEUE_NAME));
            settings.setMaxSizeBytes(1024 * 1024);
            settings.setPageSizeBytes(100 * 1024);
            settings.setAddressFullMessagePolicy(AddressFullMessagePolicy.PAGE);
            //settings.setExpiryAddress(new SimpleString("jms.queue.DLQ"));
            //settings.setRedeliveryDelay(5000);
            settings.setMaxDeliveryAttempts(3);
            server.getAddressSettingsRepository().setDefault(settings);
            server.getAddressSettingsRepository().addMatch("#", settings);
            server.start();

            ...

                      Configuration configuration = new ConfigurationImpl();

                           configuration.setPersistenceEnabled(true);

                           configuration.setSecurityEnabled(false);

                           configuration.setBindingsDirectory("D:/hq/bindings");

                           configuration.setJournalDirectory("D:/hq/journal");

                           configuration.setPagingDirectory("D:/hq/paging");

                           configuration.setJournalType(JournalType.NIO);

             

                           //configuration.set

                           configuration.getAcceptorConfigurations()

                                .add(

                                     new TransportConfiguration(InVMAcceptorFactory.class

                                          .getName()));

             

                           server = HornetQServers.newHornetQServer(configuration);

             

                           AddressSettings settings = new AddressSettings();

                           settings.setDeadLetterAddress(new SimpleString(DLQ_QUEUE_NAME));

                           settings.setMaxSizeBytes(1024 * 1024);

                           settings.setPageSizeBytes(100 * 1024);

                           settings.setAddressFullMessagePolicy(AddressFullMessagePolicy.PAGE);

                           //settings.setExpiryAddress(new SimpleString("jms.queue.DLQ"));

                           //settings.setRedeliveryDelay(5000);

                           settings.setMaxDeliveryAttempts(3);

                           server.getAddressSettingsRepository().setDefault(settings);

                           server.getAddressSettingsRepository().addMatch("#", settings);

             

                           server.start();

            ...

            • 3. Re: OutOfMemoryError on server start with "big" journal
              bodrin b. Newbie

              Anyone help?

               

              Guys, I think this can be serious problem - a bunch of messages was sent to a single durable queue, pagination is on and still the server can not start due to OutOfMemoryError ?!

               

              (I was wondering whether I can remove my MySQL backed queues and use embedded hornetq instead, but I'm afraid of this OutOfMemoryError..)

              • 4. Re: OutOfMemoryError on server start with "big" journal
                Clebert Suconic Master

                I didn't have time to go over your testcase, since we have a bunch of people on Xmas leave. (you would have support engineers also looking at the issue if this was a support case... I'm not asking you to buy support .. I'm just saying you would have extra people looking).

                 

                 

                Anyway.. from what I have seen it seems you are only applying the address-settings at the DLA. I believe your settings on paging are not being respected.

                 

                Or maybe this is related: BLOCK, DROP and PAGE policy does not work in HornetQ QA3 build  (case you're using JBoss 6)

                • 5. Re: OutOfMemoryError on server start with "big" journal
                  bodrin b. Newbie

                  10x, I can not open this link .. no permission .. but anyway I'm using hornetq embedded in my standalone app.

                   

                  I have found some examples on how to enable paging and I think my test is ok?

                   

                  Not sure which of the two lines below does the trick, so I placed both of them.

                  server.getAddressSettingsRepository().setDefault(settings);

                  server.getAddressSettingsRepository().addMatch("#", settings);

                   

                  Also can I ask you : If I create a durable queue without paging and sent a lot of messages to it, then I stop, add paging into my code and start again should it work this way?

                   

                  Another thing that I have observed is that if I create a durable queue and set DLA then if I restart my app the queue does not remember its DLA, so I presume the queues does not remember any of its configs. Is this right?

                  • 6. Re: OutOfMemoryError on server start with "big" journal
                    Clebert Suconic Master

                    I couldn't replicate any errors using your test. It all ran fine.

                     

                    We had a few issues with compacting that were fixed later. And paging is being changed on trunk At the moment. (It's done at the queue level now).

                     

                    The new paging won't prevent you from having a consumer with unacked messages what would consume all of your memory. The body of the messages are weak references when paged what will prevent OMEs but there are still space for anti-patterns or bad usages.

                     

                     

                    The bug fixed would lead to issues during compacting in such way you wouldn't be able to restart from the journal. I remember fixing an issue on the format also that would lead to an OME.

                     

                    You should probably test it on trunk.

                    • 7. Re: OutOfMemoryError on server start with "big" journal
                      Clebert Suconic Master

                      Just to clarify I tested your code against trunk and 2.1.2. NO issues found.

                      • 8. Re: OutOfMemoryError on server start with "big" journal
                        bodrin b. Newbie

                        10x for trying it Clebert ... very strange .. I am able to reproduce the above *always* using hornetq-2.1.2.Final .. after you unzip the above attachment you have an hq directory and did you changed the lines below from the test to point to it ?

                         

                                 configuration.setBindingsDirectory("D:/hq/bindings");
                                    configuration.setJournalDirectory("D:/hq/journal");
                                    configuration.setPagingDirectory("D:/hq/paging");

                         

                        I I'm interested in the answers of the questions above if someone can answer them, please, I repeat them:

                         

                        If I create a durable queue without paging and sent a lot of  messages to it, then I stop, add paging into my code and start again  should it work this way?

                         

                        Another  thing that I have observed is that if I create a durable queue and set  DLA then if I restart my app the queue does not remember its DLA, so I  presume the queues does not remember any of its configs. Is this right?

                        • 9. Re: OutOfMemoryError on server start with "big" journal
                          Clebert Suconic Master

                          Once you have damaged your journal.. it's always damaged.

                           

                          You should start your test with an empty journal, and find me a way to replicate it with an empty journal.

                           

                           

                          Anyway: As I said, I have fixed a few issues with the journal before. You should probably try with trunk.

                          • 10. Re: OutOfMemoryError on server start with "big" journal
                            Clebert Suconic Master

                            Another reason for you trying the trunk is this:

                             

                            http://community.jboss.org/wiki/Howtoreportabugissue

                             

                             

                            "We don't spend time debugging old versions of HornetQ if the bug no longer occurs in the latest version."