OutOfMemoryError on server start with "big" journal

[bodrin]

Hi,

I have found some tests here in the forum and played with them. I have a durable queue with producer (and consumer, but not included here). So I have sent some messages and then and I got OutOfMemoryError when trying to start the server .. see below.

I have turned on pagination and can't think of any other reason why this is happening, so I'm sending to you the test (two classes) and the bindings,journal and paging directories, because I have not found the exact way to reproduce it, but may be you can analyse the DB if you are interested

Best regards,

bodrin

<quote>

java.lang.OutOfMemoryError: Java heap space

at java.nio.HeapByteBuffer.<init>(Unknown Source)

at java.nio.ByteBuffer.allocate(Unknown Source)

at org.hornetq.core.journal.impl.NIOSequentialFileFactory.newBuffer(NIOSequentialFileFactory.java:80)

at org.hornetq.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:453)

at org.hornetq.core.journal.impl.JournalImpl.load(JournalImpl.java:1760)

at org.hornetq.core.journal.impl.JournalImpl.load(JournalImpl.java:1409)

at org.hornetq.core.persistence.impl.journal.JournalStorageManager.loadMessageJournal(JournalStorageManager.java:777)

at org.hornetq.core.server.impl.HornetQServerImpl.loadJournals(HornetQServerImpl.java:1220)

at org.hornetq.core.server.impl.HornetQServerImpl.initialisePart2(HornetQServerImpl.java:1070)

at org.hornetq.core.server.impl.HornetQServerImpl.start(HornetQServerImpl.java:313)

at Test.<init>(Test.java:69)

at Test.main(Test.java:232)

java.lang.Exception: Java heap space

at org.hornetq.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:764)

at org.hornetq.core.journal.impl.JournalImpl.load(JournalImpl.java:1760)

at org.hornetq.core.journal.impl.JournalImpl.load(JournalImpl.java:1409)

at org.hornetq.core.persistence.impl.journal.JournalStorageManager.loadMessageJournal(JournalStorageManager.java:777)

at org.hornetq.core.server.impl.HornetQServerImpl.loadJournals(HornetQServerImpl.java:1220)

at org.hornetq.core.server.impl.HornetQServerImpl.initialisePart2(HornetQServerImpl.java:1070)

at org.hornetq.core.server.impl.HornetQServerImpl.start(HornetQServerImpl.java:313)

at Test.<init>(Test.java:69)

at Test.main(Test.java:232)

Caused by: java.lang.OutOfMemoryError: Java heap space

at java.nio.HeapByteBuffer.<init>(Unknown Source)

at java.nio.ByteBuffer.allocate(Unknown Source)

at org.hornetq.core.journal.impl.NIOSequentialFileFactory.newBuffer(NIOSequentialFileFactory.java:80)

at org.hornetq.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:453)

... 8 more

java.lang.OutOfMemoryError: Java heap space

at java.nio.HeapByteBuffer.<init>(Unknown Source)

at java.nio.ByteBuffer.allocate(Unknown Source)

at org.hornetq.core.journal.impl.NIOSequentialFileFactory.newBuffer(NIOSequentialFileFactory.java:80)

at org.hornetq.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:453)

at org.hornetq.core.journal.impl.JournalImpl.load(JournalImpl.java:1760)

at org.hornetq.core.journal.impl.JournalImpl.load(JournalImpl.java:1409)

at org.hornetq.core.persistence.impl.journal.JournalStorageManager.loadMessageJournal(JournalStorageManager.java:777)

at org.hornetq.core.server.impl.HornetQServerImpl.loadJournals(HornetQServerImpl.java:1220)

at org.hornetq.core.server.impl.HornetQServerImpl.initialisePart2(HornetQServerImpl.java:1070)

at org.hornetq.core.server.impl.HornetQServerImpl.start(HornetQServerImpl.java:313)

at Test.<init>(Test.java:69)

at Test.main(Test.java:232)

java.lang.Exception: Java heap space

at org.hornetq.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:764)

at org.hornetq.core.journal.impl.JournalImpl.load(JournalImpl.java:1760)

at org.hornetq.core.journal.impl.JournalImpl.load(JournalImpl.java:1409)

at org.hornetq.core.persistence.impl.journal.JournalStorageManager.loadMessageJournal(JournalStorageManager.java:777)

at org.hornetq.core.server.impl.HornetQServerImpl.loadJournals(HornetQServerImpl.java:1220)

at org.hornetq.core.server.impl.HornetQServerImpl.initialisePart2(HornetQServerImpl.java:1070)

at org.hornetq.core.server.impl.HornetQServerImpl.start(HornetQServerImpl.java:313)

at Test.<init>(Test.java:69)

at Test.main(Test.java:232)

Caused by: java.lang.OutOfMemoryError: Java heap space

at java.nio.HeapByteBuffer.<init>(Unknown Source)

at java.nio.ByteBuffer.allocate(Unknown Source)

at org.hornetq.core.journal.impl.NIOSequentialFileFactory.newBuffer(NIOSequentialFileFactory.java:80)

at org.hornetq.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:453)

... 8 more

</quote>

[gaohoward]

How is your hornetq configuration like? did you set max-size-bytes?

[bodrin]

Hi, the test is into the zip attached above, but anyway here it is:

Configuration configuration = new ConfigurationImpl();

configuration.setPersistenceEnabled(true);

configuration.setSecurityEnabled(false);

configuration.setBindingsDirectory("D:/hq/bindings");

configuration.setJournalDirectory("D:/hq/journal");

configuration.setPagingDirectory("D:/hq/paging");

configuration.setJournalType(JournalType.NIO);

//configuration.set

configuration.getAcceptorConfigurations()

.add(

new TransportConfiguration(InVMAcceptorFactory.class

.getName()));

server = HornetQServers.newHornetQServer(configuration);

AddressSettings settings = new AddressSettings();

settings.setDeadLetterAddress(new SimpleString(DLQ_QUEUE_NAME));

settings.setMaxSizeBytes(1024 * 1024);

settings.setPageSizeBytes(100 * 1024);

settings.setAddressFullMessagePolicy(AddressFullMessagePolicy.PAGE);

//settings.setExpiryAddress(new SimpleString("jms.queue.DLQ"));

//settings.setRedeliveryDelay(5000);

settings.setMaxDeliveryAttempts(3);

server.getAddressSettingsRepository().setDefault(settings);

server.getAddressSettingsRepository().addMatch("#", settings);

server.start();

...

          Configuration configuration = new ConfigurationImpl();

               configuration.setPersistenceEnabled(true);

               configuration.setSecurityEnabled(false);

               configuration.setBindingsDirectory("D:/hq/bindings");

               configuration.setJournalDirectory("D:/hq/journal");

               configuration.setPagingDirectory("D:/hq/paging");

               configuration.setJournalType(JournalType.NIO);

               //configuration.set

               configuration.getAcceptorConfigurations()

                    .add(

                         new TransportConfiguration(InVMAcceptorFactory.class

                              .getName()));

               server = HornetQServers.newHornetQServer(configuration);

               AddressSettings settings = new AddressSettings();

               settings.setDeadLetterAddress(new SimpleString(DLQ_QUEUE_NAME));

               settings.setMaxSizeBytes(1024 * 1024);

               settings.setPageSizeBytes(100 * 1024);

               settings.setAddressFullMessagePolicy(AddressFullMessagePolicy.PAGE);

               //settings.setExpiryAddress(new SimpleString("jms.queue.DLQ"));

               //settings.setRedeliveryDelay(5000);

               settings.setMaxDeliveryAttempts(3);

               server.getAddressSettingsRepository().setDefault(settings);

               server.getAddressSettingsRepository().addMatch("#", settings);

               server.start();

...

[bodrin]

Anyone help?

Guys, I think this can be serious problem - a bunch of messages was sent to a single durable queue, pagination is on and still the server can not start due to OutOfMemoryError ?!

(I was wondering whether I can remove my MySQL backed queues and use embedded hornetq instead, but I'm afraid of this OutOfMemoryError..)

[clebert.suconic]

I didn't have time to go over your testcase, since we have a bunch of people on Xmas leave. (you would have support engineers also looking at the issue if this was a support case... I'm not asking you to buy support .. I'm just saying you would have extra people looking).

Anyway.. from what I have seen it seems you are only applying the address-settings at the DLA. I believe your settings on paging are not being respected.

Or maybe this is related: BLOCK, DROP and PAGE policy does not work in HornetQ QA3 build  (case you're using JBoss 6)

[bodrin]

10x, I can not open this link .. no permission .. but anyway I'm using hornetq embedded in my standalone app.

I have found some examples on how to enable paging and I think my test is ok?

Not sure which of the two lines below does the trick, so I placed both of them.

server.getAddressSettingsRepository().setDefault(settings);

server.getAddressSettingsRepository().addMatch("#", settings);

Also can I ask you : If I create a durable queue without paging and sent a lot of messages to it, then I stop, add paging into my code and start again should it work this way?

Another thing that I have observed is that if I create a durable queue and set DLA then if I restart my app the queue does not remember its DLA, so I presume the queues does not remember any of its configs. Is this right?

[clebert.suconic]

I couldn't replicate any errors using your test. It all ran fine.

We had a few issues with compacting that were fixed later. And paging is being changed on trunk At the moment. (It's done at the queue level now).

The new paging won't prevent you from having a consumer with unacked messages what would consume all of your memory. The body of the messages are weak references when paged what will prevent OMEs but there are still space for anti-patterns or bad usages.

The bug fixed would lead to issues during compacting in such way you wouldn't be able to restart from the journal. I remember fixing an issue on the format also that would lead to an OME.

You should probably test it on trunk.

[clebert.suconic]

Just to clarify I tested your code against trunk and 2.1.2. NO issues found.

[bodrin]

10x for trying it Clebert ... very strange .. I am able to reproduce the above *always* using hornetq-2.1.2.Final .. after you unzip the above attachment you have an hq directory and did you changed the lines below from the test to point to it ?

         configuration.setBindingsDirectory("D:/hq/bindings");
            configuration.setJournalDirectory("D:/hq/journal");
            configuration.setPagingDirectory("D:/hq/paging");

I I'm interested in the answers of the questions above if someone can answer them, please, I repeat them:

If I create a durable queue without paging and sent a lot of  messages to it, then I stop, add paging into my code and start again  should it work this way?

Another  thing that I have observed is that if I create a durable queue and set  DLA then if I restart my app the queue does not remember its DLA, so I  presume the queues does not remember any of its configs. Is this right?

[clebert.suconic]

Once you have damaged your journal.. it's always damaged.

You should start your test with an empty journal, and find me a way to replicate it with an empty journal.

Anyway: As I said, I have fixed a few issues with the journal before. You should probably try with trunk.

[clebert.suconic]

Another reason for you trying the trunk is this:

http://community.jboss.org/wiki/Howtoreportabugissue

"We don't spend time debugging old versions of HornetQ if the bug no longer occurs in the latest version."