messages in journal, journal gets big

[arminhaaf]

Hi

we had an OutOfMemoryError in and are currently investigating the problem

10:38:52,521 ERROR [STDERR] java.lang.OutOfMemoryError: Java heap space

10:38:52,524 ERROR [STDERR]     at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)

10:38:52,524 ERROR [STDERR]     at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)

10:38:52,525 ERROR [STDERR]     at org.hornetq.core.journal.impl.NIOSequentialFile.fill(NIOSequentialFile.java:112)

10:38:52,525 ERROR [STDERR]     at org.hornetq.core.journal.impl.JournalFilesRepository.createFile(JournalFilesRepository.java:473)

10:38:52,525 ERROR [STDERR]     at org.hornetq.core.journal.impl.JournalFilesRepository.takeFile(JournalFilesRepository.java:419)

10:38:52,528 ERROR [STDERR]     at org.hornetq.core.journal.impl.JournalFilesRepository.pushOpenedFile(JournalFilesRepository.java:386)

10:38:52,530 ERROR [STDERR]     at org.hornetq.core.journal.impl.JournalFilesRepository$1.run(JournalFilesRepository.java:342)

10:38:52,531 ERROR [STDERR]     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

10:38:52,531 ERROR [STDERR]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

10:38:52,532 ERROR [STDERR]     at java.lang.Thread.run(Thread.java:662)

maybe this is related to a big journal of about 800MB which we cannot explain, as we do not see any pending messages in any of the queues/topics

However, we used org.hornetq.core.persistence.impl.journal.PrintData to show the content of the journal and can see about 100000 messages in the journal. If I interpret the data correct, the messages are for (2) topic subscriptions, which are no longer subscribed, resp. we cannot see any subscription via the jmx management. As we cannot see any subscriptions, we cannot delete the messages.

A solution is to restart hornetq and drop the journal. But you know, we will loose all other messages also (we can now live with the loss, but later on this is a problem).

So are there more sophisticated methods

- to check if there are pending messages on the subscription

- the cleanup the journal resp. to remove the messages

we are using hornet 2.2.2 with jboss 5.1. Only hornetq is running on jboss...

Thanx

Armin

[clebert.suconic]

Don't you see the Queues on the bindings when using PrintData?

PrintData will be divided into two steps. I - will print the adds and deletes. II - Will print only surviving records.

I'm always interested in broken data to avoid any issues for other users. If you send it to me I could take a look.

(records are just alleatory data for me, but I wouldn't get it if that's something that would create me lliability).

[arminhaaf]

In fact, i do not fully understand the journal data.  But I interpreted the surving records as messages that are not yet delivered, i hope I was correct.

I do not know if the printdata is incorrect, or I'm not able to see the messages anywhere in the jmx console. But I think the PrintData is correct, as we have a big heap allocation after startup of jboss and the journal has always the same footprint in the filesystem.

Which data do you need? The whole hornetq data directory (about 1GB unzipped) or the result of the PrintData (6MB zipped)

[clebert.suconic]

The whole data... you will have to send me an email probably

You will probably have to place it on some http server (I doubt it will fit an email even zipped)

csuconic

at

redhat

[clebert.suconic]

Maybe you deleted the JMS binding but not the core-queue.

I - make sure you delete the core queues associated with your addresses.

II - also configure paging or blocking so you won't get to a scenario where you have more data than it fits on the JVM.

III - process your data with a bit of more memory, then limit your data later.

(You sent me your data in pvt and I didn't see anything damaged).